The Word Emission Latency Problem in Voice Agents
In full-duplex conversational voice AI, overall pipeline latency is strictly bounded: the entire round-trip—comprising Automatic Speech Recognition (ASR), LLM generation, and Text-to-Speech (TTS) synthesis—must complete in under 400 milliseconds. While modern inference engines achieve sub-40ms Time to First Token (TTFT) with Continuous Batching & PagedAttention, legacy ASR architectures remain the single largest contributor to user-perceived hesitation.
Standard offline models like OpenAI Whisper process audio in 30-second window blocks using non-causal bidirectional self-attention across the entire mel-spectrogram. Adapting these transformer architectures for streaming telephony requires slicing incoming 16kHz audio into small causal chunks. However, naive chunking introduces an acute trade-off: short chunks emit text faster but lack right-context acoustic frames, spiking the Word Error Rate (WER) on co-articulated phonemes and triggering phantom hallucinations on background noise. Conversely, larger chunks improve phonetic confidence at the expense of human conversational cadence.
1. Acoustic Frame Chunk Sizing vs. Word Error Rate (WER)
To identify the optimal trade-off point between transcription accuracy and word emission latency, we evaluated a streaming Conformer-CTC model against LocalAgreement-2 streaming Whisper variants over 2,000 real-world customer service calls across clean WebRTC and 8kHz G.711 SIP trunks:
| Chunk Sizing (ms) | Lookahead Context | Word Emission Latency (p90) | WER (Clean 16kHz Audio) | WER (Telephony 8kHz G.711) |
|---|---|---|---|---|
40ms (640 samples) |
0ms (Strictly Causal) | 68ms | 18.4% | 27.9% (Unstable) |
80ms (1,280 samples) |
40ms Lookahead (2 frames) | 122ms | 7.8% | 11.2% |
160ms (2,560 samples) |
80ms Lookahead (4 frames) | 210ms | 4.9% | 6.8% |
320ms (5,120 samples) |
160ms Lookahead (8 frames) | 395ms | 4.1% | 5.4% |
As benchmarked above, an 80ms causal chunk with a 40ms lookahead context hits the sweet spot for real-time voice agents: it achieves a p90 word emission latency of 122ms while keeping Word Error Rates within acceptable production thresholds on degraded telephony codecs.
2. Zero-Allocation Streaming Audio Ring Buffer
In a production voice gateway, incoming audio arrives over WebSockets or RTP at 20ms packet intervals as raw 16-bit linear PCM audio. Slicing must happen in user space with zero heap allocations inside the event loop using pre-allocated ring buffers:
# asr/streaming_buffer.py
import numpy as np
class AcousticRingBuffer:
"""Zero-allocation audio buffer designed for causal chunking with lookahead."""
def __init__(self, chunk_size_ms: int = 80, lookahead_ms: int = 40, sample_rate: int = 16000):
self.chunk_samples = int(sample_rate * (chunk_size_ms / 1000.0))
self.lookahead_samples = int(sample_rate * (lookahead_ms / 1000.0))
self.total_window_samples = self.chunk_samples + self.lookahead_samples
self.capacity = self.total_window_samples * 16
self.buffer = np.zeros(self.capacity, dtype=np.float32)
self.write_idx = 0
self.read_idx = 0
def append_pcm16(self, raw_bytes: bytes) -> None:
"""Convert raw PCM16 bytes directly into float32 normalized samples."""
int16_samples = np.frombuffer(raw_bytes, dtype=np.int16)
samples = int16_samples.astype(np.float32) / 32768.0
n = len(samples)
# Compact buffer if approaching capacity boundary
if self.write_idx + n > self.capacity:
unread = self.write_idx - self.read_idx
self.buffer[:unread] = self.buffer[self.read_idx:self.write_idx]
self.read_idx = 0
self.write_idx = unread
self.buffer[self.write_idx:self.write_idx + n] = samples
self.write_idx += n
def pop_frame_with_lookahead(self) -> np.ndarray | None:
"""Emit next chunk with required lookahead if enough audio has buffered."""
available = self.write_idx - self.read_idx
if available < self.total_window_samples:
return None
frame = self.buffer[self.read_idx:self.read_idx + self.total_window_samples]
self.read_idx += self.chunk_samples # Advance only by chunk_samples, preserving lookahead
return frame
3. CTC Blank-Token Decoding, LocalAgreement-2 & Hallucination Suppression
Streaming ASR models suffer from two distinct production failure modes: unstable partial token flickering at chunk boundaries and phantom hallucinations ("Thank you for watching", "Subtitles by...") when acoustic energy dips into ambient noise. To guarantee deterministic transcription stability:
- CTC Prefix Beam Search with Blank Thresholding: In Conformer-CTC pipelines, suppress non-blank token emission whenever the posterior probability of the CTC blank token
P(∅)exceeds0.85across three consecutive 20ms frames. - LocalAgreement-2 Prefix Locking: For encoder-decoder Whisper streaming, only commit tokens to the downstream LLM prompt when two consecutive sliding acoustic windows produce identical token prefixes, eliminating mid-word backtracking.
- Pre-ASR Silero VAD Gating: Gate the acoustic ring buffer with an ONNX Silero VAD filter so background line hum never enters the mel-spectrogram extractor.
# asr/ctc_stability_gate.py
from dataclasses import dataclass
@dataclass
class TranscriptCommitState:
committed_tokens: list[int]
previous_hypothesis: list[int]
def stabilize_streaming_hypothesis(
state: TranscriptCommitState,
current_tokens: list[int],
blank_prob: float,
rms_energy_db: float,
) -> list[int]:
"""Commits only stable LocalAgreement-2 prefix tokens above acoustic floor."""
if rms_energy_db < -48.0 or blank_prob > 0.88:
return []
common_prefix_len = 0
for prev_tok, curr_tok in zip(state.previous_hypothesis, current_tokens):
if prev_tok == curr_tok:
common_prefix_len += 1
else:
break
stable_prefix = current_tokens[:common_prefix_len]
newly_committed = stable_prefix[len(state.committed_tokens):]
state.committed_tokens.extend(newly_committed)
state.previous_hypothesis = current_tokens
return newly_committed
For related production architectures and system implementations, explore these companion guides:
- Sub-50ms Speech-to-Text Pipeline with Silero VAD — Combine causal acoustic lookahead frames with ONNX Silero VAD and WebSocket backpressure queues.
- WebRTC TWCC & Adaptive Bitrate in Voice Telephony — Maintain clean acoustic frames across fluctuating mobile links with RFC 8888 congestion control.
- Speech-to-Speech (S2S) vs. Cascaded STT-LLM-TTS Pipelines — Compare streaming Conformer/Whisper ASR cascades against native speech-to-speech architectures.
- Real-Time Voice AI & Telephony Engineering — Explore our production engineering deliverables, or benchmark your stack with the interactive Voice AI Latency & Cost Forecaster.
Key Architectural Takeaways
- Size lookahead frames explicitly: An 80ms causal chunk paired with a 40ms lookahead window reduces WER by more than 55% compared to strictly causal 40ms chunks while holding p90 emission latency under 125ms.
- Eliminate GC churn in audio loops: Pre-allocate contiguous NumPy ring buffers to process 16kHz PCM frames without triggering Python garbage collection pauses.
- Pair VAD gating with prefix stability: Use LocalAgreement-2 prefix locking and our Voice AI Latency Forecaster to budget end-to-end turn-taking latency. For full telephony architectures, review our Voice AI Telephony Platform Case Study.