Continuous Batching & PagedAttention in Real-Time Voice AI: Slashing Token Inter-Arrival Time Below 50ms

The Strict Latency Budget of Conversational Voice Agents

In conversational voice AI architectures, the human ear perceives any round-trip acoustic pause greater than 300 to 500 milliseconds as an unnatural, halting conversation. In a standard full-duplex voice pipeline comprising Automatic Speech Recognition (ASR), Large Language Model (LLM) Inference, and Text-to-Speech (TTS) synthesis, the latency budget allocation is unforgiving:

• ASR Endpointing & Transcription: 80–120ms
• LLM Time-To-First-Token (TTFT): 80–120ms
• LLM Token Inter-Arrival Time (TBT / Streaming Jitter): < 40ms per token
• Streaming TTS Buffer & First Audio Chunk Framing: 80–120ms

If an LLM inference engine suffers from queuing latency or irregular token inter-arrival times, downstream streaming TTS buffers underrun, generating audible glitching, stuttering, or awkward conversational pauses.

The Failure of Static & Naive Dynamic Batching

Traditional deep learning model serving (such as TorchServe or early Triton deployments) relies on static or time-windowed dynamic batching. In these paradigms, requests are grouped at the sequence level:

1. The engine waits for up to N requests or until a batch timeout expires (e.g., 20ms).
2. All sequences in the batch are padded with dummy tokens to match the length of the longest request in the batch.
3. The GPU processes the entire batch through prefill and autoregressive decoding steps until all sequences hit their respective end-of-sequence (EOS) tokens.

In multi-tenant conversational voice AI, dialog turns are wildly heterogeneous: User A asks a concise question ("What's the weather in Nairobi?"), requiring a 20-token reply, while User B requests an extensive explanation requiring 300 tokens. Under static batching, User A's short response remains trapped in GPU compute iterations until User B's long response completes, driving tail p99 latency through the roof.

Iteration-Level Continuous Batching

To eliminate conversational jitter, modern voice inference runtimes (such as vLLM and TensorRT-LLM) implement Continuous Batching (also termed iteration-level scheduling). Rather than binding requests to a static batch lifecycle, the scheduler operates at the granularity of individual forward passes:

Iteration 1: [Seq A: Token 1] [Seq B: Token 4] [Seq C: Prefill Phase]
Iteration 2: [Seq A: Token 2] [Seq B: Token 5] [Seq D: Token 1 (New Incoming!)]
Iteration 3: [Seq A: EOS Done!] [Seq B: Token 6] [Seq D: Token 2] -> Seq A Dispatched to TTS!
Iteration 4: [Seq E: Prefill]  [Seq B: Token 7] [Seq D: Token 3]

As soon as a sequence generates an end-of-thought token or punctuation delimiter, it is immediately ejected from the GPU decoding iteration, freeing tensor memory and delivering streaming tokens to the TTS pipeline with zero artificial holding delay.

PagedAttention: Eliminating Physical KV Cache Fragmentation

Continuous batching solves compute scheduling, but creates a severe physical memory crisis in GPU High Bandwidth Memory (HBM): KV Cache Fragmentation. In standard Transformer architectures, key-value vectors for past tokens are cached in contiguous GPU memory. Because the engine cannot predict how many tokens a user conversation will generate, it must pre-allocate contiguous memory for the maximum possible context length (e.g., 2,048 or 8,192 tokens).

In production, this leads to 60% to 80% wasted GPU memory due to internal and external fragmentation:

• Internal Fragmentation: Reserved memory slots that are never utilized during short turns.
• External Fragmentation: Memory gaps between sequences that are too small to allocate for incoming requests.

Inspired by virtual memory paging in operating systems, PagedAttention divides the KV cache into fixed-size physical blocks (e.g., 16 or 32 tokens per block). A software Block Table maps logical sequence token indices to arbitrary physical blocks scattered across GPU VRAM:

# Logical Representation in PagedAttention
# Sequence Turn: "The quick brown fox jumps over the lazy dog"
# Block Size: 4 tokens

Logical Blocks:
  Block 0: ["The", "quick", "brown", "fox"]
  Block 1: ["jumps", "over", "the", "lazy"]
  Block 2: ["dog"]

Physical GPU Block Table:
  Block 0 -> Physical Frame @ 0x7f9a1400 (VRAM Block 12)
  Block 1 -> Physical Frame @ 0x7f98e200 (VRAM Block 84)
  Block 2 -> Physical Frame @ 0x7f9b0000 (VRAM Block 3)

During the attention computation, specialized CUDA kernels read non-contiguous physical blocks seamlessly using pointer dereferencing, slashing memory waste to under 4%.

Production Engine Tuning: Sizing vLLM for Sub-50ms Voice Streams

Below is a battle-tested vLLM production launch configuration optimized specifically for low-latency, multi-tenant voice dialog workloads on NVIDIA A100/H100 GPUs:

python3 -m vllm.entrypoints.openai.api_server   --model meta-llama/Llama-3.1-8B-Instruct   --tensor-parallel-size 1   --gpu-memory-utilization 0.92   --max-model-len 4096   --block-size 16   --max-num-seqs 64   --max-num-batched-tokens 2048   --scheduling-policy priority   --enable-prefix-caching   --disable-log-requests   --port 8000

Key configuration parameters calibrated for voice AI:

• --block-size 16: Sizing blocks to 16 tokens minimizes tail waste on short conversational voice turns compared to default 32-token blocks.
• --max-num-batched-tokens 2048: Caps the prefill chunk size, ensuring that a large incoming context prefill never stalls concurrent active decoding iterations for longer than 35ms.
• --enable-prefix-caching: Automatically caches system prompts, voice persona guardrails, and tool definitions across multi-turn dialogs, achieving sub-40ms TTFT on subsequent turns.

Empirical Benchmarks: Static vs. Continuous PagedAttention

Across a stress test simulating 32 concurrent full-duplex conversational sessions with bursty turn intervals:

• Static Batching: TTFT p99: 480ms; Token Inter-Arrival Time (TBT): 112ms; GPU Memory Saturation: 96% (OOM crashes at 38 streams).
• Continuous Batching with PagedAttention: TTFT p99: 84ms; Token Inter-Arrival Time (TBT): 31ms; GPU Memory Saturation: 68% (zero crashes, sustained up to 74 concurrent streams).

For organizations deploying enterprise telephony agents and speech infrastructure, exploring our Real-Time Voice AI Pipeline Architecture reveals how we integrate WebRTC media gateways with continuous batching backends.