The Fundamental Limit of Cascaded Voice Architectures
Over the past four years, conversational voice AI has predominantly relied on a cascaded tripartite architecture. Incoming user speech is processed in three distinct sequential steps:
User Audio ---> [ 1. ASR / Whisper ] ---> Text Tokens ---> [ 2. LLM / GPT-4 ] ---> Text Stream ---> [ 3. TTS / ElevenLabs ] ---> Output Audio
While engineering teams have heavily optimized individual components—using streaming ASR lookahead chunking, speculative decoding, and streaming WebSocket audio—the cascaded paradigm faces an insurmountable barrier: compounding serialization latency.
| Pipeline Stage | Minimum Realistic Production Latency | Average Real-World Telephony Latency |
|---|---|---|
| Audio Capture & Acoustic VAD Frame | 60 ms | 120 ms |
| ASR Word Finalization | 120 ms | 180 ms |
| LLM Time to First Token (TTFT) | 140 ms | 250 ms |
| TTS Sentence Boundary Chunking | 180 ms | 300 ms |
| Network Jitter Buffer & WebRTC Playback | 40 ms | 80 ms |
| Total End-to-End Turnaround | 540 ms | 930 ms (Noticeable Hesitation) |
1. The Information Loss Problem: Text as an Inadequate Bottleneck
Beyond latency, serializing speech into plain text discards critical human communicative information. Written text strips away pitch, cadence, hesitation, sarcastic inflection, whisper tones, and background emotional distress. If a caller sighs heavily or speaks with urgency, an ASR model produces the same flat string: "I need help right now."
Similarly, text-to-speech synthesis must guess the appropriate prosody from bare punctuation marks, frequently producing robotic or emotionally mismatched intonations.
2. The Speech-to-Speech (S2S) Architecture
Next-generation systems—such as Kyutai's Moshi, Mini-Omni, and OpenAI's GPT-4o Voice—eliminate text as the intermediate transport medium. Instead, these models operate as Audio-In / Audio-Out native multimodal transformers.
User Audio Stream ---> [ Neural Audio Codec (SNAC / EnCodec) ] ---> Acoustic Tokens ---> [ Dual-Stream S2S Transformer ] ---> Output Acoustic Tokens ---> [ Neural Vocoder ] ---> Audio Stream
3. Discrete Neural Audio Tokenization
Native S2S models convert continuous 24kHz audio waveforms into discrete tokens using Residual Vector Quantization (RVQ) codecs such as EnCodec, Mimi, or SoundStream. A single 20ms audio frame is compressed into 8 hierarchical codebook levels, representing both phonetic semantic content (lower levels) and fine acoustic prosody/speaker identity (higher levels).
# audio/s2s_gateway.py
import asyncio
import numpy as np
class SpeechToSpeechStreamer:
"""Simultaneous full-duplex audio token streaming gateway."""
def __init__(self, s2s_client, codec_decoder):
self.client = s2s_client
self.decoder = codec_decoder
self.audio_out_queue = asyncio.Queue()
async def ingest_audio_chunks(self, pcm_stream):
"""Streams 20ms audio frames directly into the neural audio tokenizer."""
async for frame in pcm_stream:
# Codec encodes 20ms of audio (320 samples at 16kHz) into discrete tokens
acoustic_tokens = self.decoder.encode_frame(frame)
# S2S model processes audio tokens and yields response tokens simultaneously
async for out_token in self.client.generate_stream(acoustic_tokens):
synthesized_pcm = self.decoder.decode_token(out_token)
await self.audio_out_queue.put(synthesized_pcm)
4. Architectural Comparison: Cascaded vs. Native S2S
| Engineering Dimension | Cascaded (STT → LLM → TTS) | Native Speech-to-Speech (S2S) |
|---|---|---|
| Conversational Turnaround | 650ms to 1,200ms | 180ms to 280ms (Human Parity) |
| Prosody & Emotion Preservation | Zero (Stripped by text normalization) | Native (Acoustic token cross-attention) |
| Full-Duplex Interruption (Barge-In) | Requires external VAD halting loops | Native (Model listens while speaking) |
| Compute & Tool Calling Complexity | Easy (Standard JSON structured outputs) | Challenging (Requires parallel text heads) |
While native S2S models present new challenges for deterministic tool execution and safety guardrails, they fundamentally redefine conversational latency, delivering sub-300ms turnaround with human-level expressive prosody. Learn more about telephony pipelines in our guide to WebRTC TWCC Congestion Control.
Production Latency Budgeting, Guardrails & Telephony Telemetry
Deploying End-to-End Speech-to-Speech (S2S) Models vs. Cascaded STT-LLM-TTS: Latency Budgets & Prosody in Voice AI in a commercial conversational AI or real-time voice telephony pipeline requires strict adherence to deterministic latency and reliability budgets. Unlike asynchronous chat interfaces where users tolerate a 2-to-3 second pause, full-duplex voice and real-time inference pipelines collapse whenever end-to-end turn-around latency exceeds 500 to 700 milliseconds at the 95th percentile (p95).
To ensure consistent sub-second performance across thousands of concurrent sessions, production deployments must instrument three operational safeguards:
- Per-Stage Percentile Telemetry (TTFT & TTFB): Log independent histograms for acoustic frame ingestion, Voice Activity Detection (VAD) endpointing, LLM Time-to-First-Token (TTFT), and Text-to-Speech (TTS) Time-to-First-Byte (TTFB). A 200ms regression in any single microservice immediately degrades conversational prosody and barge-in responsiveness.
- Deterministic Circuit Breakers & Provider Failover: When upstream inference APIs or GPU workers experience queue congestion, enforce a strict 650ms first-token timeout that seamlessly fails over to a secondary warm regional endpoint or quantized fallback model without dropping the caller's WebRTC or SIP media session.
- Memory & Socket Lifecycle Hygiene: Bound per-session audio ring buffers and KV-cache allocations explicitly so long-running 30-minute conversations never trigger memory fragmentation or event-loop stalls.
How can we benchmark the unit economics and latency of this voice AI architecture before launch?
Model your concurrent call volume and token throughput using our interactive Voice AI Cost & Latency Calculator and Audio Latency Tester, or book a Voice AI Engineering Discovery Session with our systems team.
For related production architectures and system implementations, explore these companion guides:
- Acoustic Filler Audio & Interruption Handling in Voice AI — Eliminate perceived turn-taking latency in cascaded and speech-to-speech telephony pipelines.
- The True Cost of Building an Enterprise AI Voice Agent — Compare per-minute unit economics between native Speech-to-Speech APIs and modular STT-LLM-TTS stacks.
- Streaming Speech Recognition with Conformer & Whisper — Optimize the ASR front-end of cascaded voice agents to rival native S2S response times.
- Real-Time Voice AI & Telephony Engineering — Explore our production engineering deliverables, or benchmark your stack with the interactive Voice AI Latency & Cost Forecaster.