Context Compaction & State Re-Anchoring: Maintaining Long-Lived AI Voice Agent Conversations

The Long-Call Dilemma: TTFT Creep in Real-Time Voice

Most benchmarks for conversational voice agents assume brief interactions: a 2-minute weather check or a 3-turn flight lookup. In real-world enterprise deployments (such as technical support, insurance claims intake, or healthcare consultations), telephony calls routinely extend between 15 and 45 minutes.

As dialogue progresses, conversational history accumulates rapidly: transcripts, function-calling arguments, tool responses, and agent utterances swell the LLM context window to 4,000+ tokens. This accumulation creates an acute production bottleneck: Time to First Token (TTFT) degradation.

During the first minute of a call, the LLM processes a 400-token prompt and responds with a TTFT of 180 milliseconds. By minute 25, the model must process a 4,500-token prompt on every turn. Because prefill attention compute scales with context length, TTFT drifts upward from 180ms to over 850 milliseconds. Combined with Speech-to-Text (STT) and Text-to-Speech (TTS) delays, total conversational turnaround exceeds 1,400ms—violating human conversational pacing and triggering accidental user interruptions.

1. Dual-Track Architecture: Foreground Dialogue vs. Background Compaction

To preserve deterministic sub-200ms latency across indefinite call durations, we decouple conversational memory into two parallel asynchronous execution tracks:

• Track 1: Foreground Real-Time Loop (The Hot Path): Manages the live WebRTC audio stream, client VAD, STT transcription, and passes only a minimal dialogue window (the system prompt plus the last 3 conversational turns) to the primary LLM. This guarantees that prompt prefill length never exceeds 600 tokens, maintaining flat 160ms TTFT permanently.
• Track 2: Background Context Compactor (The Cold Path): Operates asynchronously while the agent is speaking synthesized audio. Every 4 turns, a lightweight background LLM (e.g. Llama-3-8B-Instruct or Claude Haiku) scans historical turns, extracts essential entities and decisions, updates an Episodic State Vector, and commits the state to an in-memory Redis session cache.

[User Speaks] ---> [STT Engine] ---> [Foreground Dialogue Manager]
                                          |
                        +-----------------+-----------------+
                        |                                   |
                [Hot Path: Last 3 Turns]             [Background Queue]
                        |                                   |
                [Primary Fast LLM]                  [Compactor LLM (Worker)]
                        |                                   |
                [Streaming Neural TTS]              [Episodic State Vector]
                        |                                   |
                [Audio to WebRTC]                   [Redis Session Store]
                        |                                   |
        (Agent speaks for 4 seconds!) ---> [Re-anchor System Prompt!]

2. Structuring the Episodic State Vector

Rather than summarizing dialogue into ambiguous prose (which causes hallucinations and detail loss), the background compactor extracts facts into a strict, validated JSON schema:

{
  "call_session_id": "voc_9841_axf",
  "caller_profile": {
    "account_number": "ACC-849201",
    "authenticated": true,
    "user_sentiment": "frustrated_billing"
  },
  "verified_facts": {
    "billing_period": "September 2026",
    "disputed_charge_amount": 149.50,
    "disputed_service": "Cloud Storage Tier 2"
  },
  "action_log": [
    {"action": "lookup_invoice", "status": "COMPLETED", "result_id": "INV-1092"},
    {"action": "waive_late_fee", "status": "APPROVED", "amount": 15.00}
  ],
  "open_topics": [
    "Credit card refund vs. account credit preference"
  ]
}

3. State Re-Anchoring Without Prompt Race Conditions

The primary technical hazard in background context compaction is prompt race conditions. If the system prompt is replaced while the user is actively speaking or while the primary LLM is generating a response, conversational context fractures.

To execute seamless prompt re-anchoring, the system updates the active session prompt exclusively during the Agent Speaking Window. When the TTS engine begins streaming synthesized audio to the caller, the agent is guaranteed to hold the conversational floor for several seconds. The dialogue orchestrator atomically swaps the system prompt with the newly compacted Episodic State Vector.

4. Production Blueprint: Python Asyncio Implementation

The following production engine coordinates foreground dialogue execution with background compaction using Python asyncio and Redis:

import asyncio
import json
import time
from typing import List, Dict

class ConversationalContextManager:
    def __init__(self, session_id: str, compactor_client, max_active_turns: int = 4):
        self.session_id = session_id
        self.compactor = compactor_client
        self.max_active_turns = max_active_turns
        self.active_turns: List[Dict[str, str]] = []
        self.episodic_state: Dict = {}
        self.compaction_task: asyncio.Task = None
        self.lock = asyncio.Lock()

    async def add_turn(self, role: str, content: str):
        async with self.lock:
            self.active_turns.append({"role": role, "content": content, "timestamp": time.time()})
            
            # Check if compaction threshold exceeded and no compactor is currently running
            if len(self.active_turns) >= (self.max_active_turns * 2) and (
                self.compaction_task is None or self.compaction_task.done()
            ):
                # Slice historical turns to compact, keeping the most recent 3 turns in active memory
                turns_to_compact = self.active_turns[:-3]
                self.compaction_task = asyncio.create_task(self._run_compaction(turns_to_compact))

    async def _run_compaction(self, turns_to_compact: List[Dict[str, str]]):
        """Runs asynchronously in the background while user/agent are conversing."""
        try:
            prompt = (
                "Analyze the following conversation turns and update the episodic state JSON.\n"
                f"Current State: {json.dumps(self.episodic_state)}\n"
                f"New Turns: {json.dumps(turns_to_compact)}\n"
                "Output ONLY valid JSON matching the Episodic State Schema."
            )
            new_state_json = await self.compactor.complete(prompt)
            updated_state = json.loads(new_state_json)

            async with self.lock:
                self.episodic_state = updated_state
                # Evict compacted turns from foreground memory
                compacted_timestamps = {t["timestamp"] for t in turns_to_compact}
                self.active_turns = [
                    t for t in self.active_turns if t["timestamp"] not in compacted_timestamps
                ]
        except Exception as err:
            # On compactor failure, foreground dialogue continues uninterrupted
            print(f"Context compaction warning: {err}")

    def build_llm_messages(self, base_system_instruction: str) -> List[Dict[str, str]]:
        """Constructs an ultralight prompt for the foreground real-time LLM."""
        state_injection = f"\nCURRENT VERIFIED EPISODIC STATE:\n{json.dumps(self.episodic_state, indent=2)}"
        system_content = f"{base_system_instruction}\n{state_injection}"

        messages = [{"role": "system", "content": system_content}]
        messages.extend([{"role": t["role"], "content": t["content"]} for t in self.active_turns])
        return messages

5. Latency Stability Across 45-Minute Telephony Calls

In stress testing simulating 45-minute continuous customer service dialogues (over 120 conversational turns):

By delegating context summarization to an out-of-band asynchronous worker and anchoring verified state into compact JSON schemas, voice agents sustain fluid conversational cadence indefinitely regardless of call duration.