Sub-50ms Speech-to-Text Pipeline: Streaming Whisper Chunking, Silero VAD, and TCP Backpressure

The Latency Anatomy of Conversational Voice AI

In conversational voice AI applications, the difference between a natural, human-like dialogue and an awkward, robotic experience is determined by a single threshold: End-to-End Latency. Human conversation operates with an average inter-turn gap of approximately 200 to 300 milliseconds. Once system response latency exceeds 600ms, users perceive noticeable lag, leading to accidental speech collisions and conversational breakdown.

The total latency budget comprises four discrete pipeline stages:

Total Latency = Audio Ingest & VAD (50ms) + ASR Transcription (120ms) + LLM First-Token TTFT (150ms) + TTS Synthesis & Audio Playback (80ms) ≈ 400ms

In traditional setups, the primary bottleneck is naive Speech-to-Text (ASR). If the system waits for the user to completely finish speaking and detect 1,000ms of trailing silence before sending the audio to an ASR model, your latency budget is instantly blown before the LLM has generated a single token.

1. The Streaming Pipeline Architecture

To break the latency barrier, real-time voice architectures must employ Streaming Speech Ingestion:

2. Instantaneous Endpointing with Silero VAD

Traditional energy-based threshold detectors fail in noisy environments (e.g. coffee shops, keyboard clicks, breathing into a microphone). Silero VAD uses a lightweight recurrent neural network (RNN) running on ONNX Runtime that outputs a precise speech probability score (0.0 to 1.0) for every 32ms audio frame:

import numpy as np
import onnxruntime

class StreamingVAD:
    def __init__(self, model_path="silero_vad.onnx", threshold=0.5):
        opts = onnxruntime.SessionOptions()
        opts.inter_op_num_threads = 1
        opts.intra_op_num_threads = 1
        self.session = onnxruntime.InferenceSession(model_path, opts, providers=['CPUExecutionProvider'])
        self.threshold = threshold
        self.state = np.zeros((2, 1, 128), dtype=np.float32)
        self.sr = np.array(16000, dtype=np.int64)

    def process_frame(self, audio_chunk_512: np.ndarray) -> bool:
        """Process 512 samples (32ms of 16kHz mono audio). Returns True if speech detected."""
        ort_inputs = {
            'input': audio_chunk_512.reshape(1, -1),
            'state': self.state,
            'sr': self.sr
        }
        out, self.state = self.session.run(None, ort_inputs)
        speech_prob = out[0][0]
        return speech_prob > self.threshold

By monitoring speech probabilities, the system detects when a user begins speaking within 32 milliseconds, immediately sending an interrupt signal to stop any ongoing text-to-speech audio playback.

3. Sliding Audio Buffers and Boundary Clipping Mitigation

Because Whisper was trained on 30-second audio chunks, passing very small raw fragments (e.g. 200ms) leads to hallucinated tokens or clipped word boundaries. To achieve accurate transcription without waiting for sentences to finish, implement an Overlapping Sliding Audio Ring Buffer:

import numpy as np

class AudioRingBuffer:
    def __init__(self, sample_rate=16000, max_seconds=3.0, window_step_seconds=0.5):
        self.sample_rate = sample_rate
        self.max_samples = int(sample_rate * max_seconds)
        self.step_samples = int(sample_rate * window_step_seconds)
        self.buffer = np.zeros(self.max_samples, dtype=np.float32)
        self.pointer = 0

    def append(self, pcm_chunk: np.ndarray):
        num_samples = len(pcm_chunk)
        if self.pointer + num_samples <= self.max_samples:
            self.buffer[self.pointer:self.pointer + num_samples] = pcm_chunk
            self.pointer += num_samples
        else:
            # Shift buffer left by step_samples to maintain rolling window
            shift = num_samples
            self.buffer[:-shift] = self.buffer[shift:]
            self.buffer[-shift:] = pcm_chunk

    def get_window(self) -> np.ndarray:
        return self.buffer[:self.pointer].copy()

Every 250 milliseconds, the ASR worker transcribes the active buffer window using faster-whisper with beam_size=1 and temperature=0.0. By feeding previous transcriptions as the initial_prompt, Whisper maintains language model context across chunk boundaries, eliminating repeated words or dropped phonemes.

4. WebSocket Backpressure and TCP Buffer Management

Under fluctuating mobile network conditions or intermittent GPU inference contention, incoming audio frames may arrive faster than the ASR engine can process them. If unmanaged, raw audio packets accumulate in memory queues, introducing artificial latency where the transcription lags 2 to 3 seconds behind the real-time speaker.

To eliminate latency buildup, the pipeline must enforce strict Backpressure & Frame Dropping Mechanics:

import asyncio

class BackpressureAudioQueue:
    def __init__(self, maxsize=15): # Max ~300ms of buffered audio
        self.queue = asyncio.Queue(maxsize=maxsize)

    async def put(self, audio_frame):
        if self.queue.full():
            # Queue saturated! Drop oldest un-inferred frame to prioritize real-time freshness!
            try:
                self.queue.get_nowait()
            except asyncio.QueueEmpty:
                pass
        await self.queue.put(audio_frame)

By enforcing a bounded queue depth, the system guarantees that transcription workers always operate on the most recent, freshest speech tokens, preventing conversational drift.

5. Production Optimization Takeaways

• Quantize to INT8 via CTranslate2: Never run raw PyTorch FP32 models in low-latency voice pipelines. INT8 quantization cuts GPU memory bandwidth by 75% and slashes inference latency by 3x with zero measurable drop in word error rate (WER).
• Pin VAD to CPU Cores: Run Silero VAD on host CPU threads using ONNX Runtime. Keeping VAD off the GPU leaves CUDA cores entirely unencumbered for Whisper inference and LLM token generation.

By combining sub-millisecond Silero VAD, sliding overlapping ring buffers, and bounded WebSocket queues, engineering teams achieve true sub-50ms speech ingestion, setting the foundation for seamless, natural real-time voice interactions.