Sub-Second Audio Chunking & Turn Detection for Real-Time LLM Voice: Beyond Fixed-Buffer Silence Windows

UI Integration: When streaming bidirectional audio and conversational transcripts to user interfaces, learn how to handle high-frequency real-time UI in React by decoupling WebSockets from component renders.

The Conversational Latency Dilemma in Voice AI

Building real-time conversational voice AI agents requires orchestrating speech-to-text (STT), large language models (LLMs), and text-to-speech (TTS) engines across persistent WebRTC or WebSocket connections. While modern cloud inference allows streaming TTS within 200ms of the first token, human conversations feel broken if the initial turn-detection latency exceeds 500ms. In natural human discourse, the gap between speaking turns is typically between 200ms and 350ms.

Most naive voice implementations rely on a fixed silence window: the system waits until the microphone stream reports 500ms to 800ms of continuous silence before declaring that the user has stopped speaking. If set too high (e.g., 750ms), the conversational cadence feels sluggish, robotic, and unresponsive. If set too low (e.g., 250ms), the agent rudely interrupts the user whenever they pause to take a breath or formulate a thought.

1. Multi-Stage Ingestion Pipeline Architecture

To achieve fluid, sub-300ms turn detection without premature interruptions, enterprise voice architectures deploy a multi-layered acoustic and semantic pipeline:

1. Frame-Level Neural VAD: Inbound raw PCM audio (16kHz, 16-bit mono) is chunked into ultra-small 20ms or 30ms frames (512 samples) and evaluated by a lightweight neural Voice Activity Detector (such as Silero VAD) executing on CPU in sub-millisecond inference time.
2. Adaptive Energy & Noise Thresholding: The detector dynamically tracks ambient noise floors, preventing keyboard clicks, air conditioner hum, or background office chatter from keeping the speech state active.
3. Semantic Endpointing: When acoustic silence exceeds 200ms, the system streams the interim STT transcript to a fast speculative classifier (or token probability model) that determines whether the utterance represents a grammatically complete thought or a suspended sentence (e.g., "I wanted to book a flight to..." vs. "I wanted to book a flight to London.").
4. Playback Queue Preemption: If speech is detected while the AI agent is actively playing outbound audio, an immediate cancellation signal flushes audio buffers on both server and client within 40ms.

2. Comparing Turn-Detection Paradigms

The operational trade-offs between fixed silence windows, neural frame analysis, and hybrid semantic endpointing are detailed below:

3. Python Implementation: Async Audio Chunking & VAD State Machine

Below is a production-ready asynchronous Python processor utilizing Silero VAD over incoming audio frames:

import asyncio
import numpy as np
import torch
import logging

logger = logging.getLogger("voice-vad")

class AudioTurnDetector:
    def __init__(self, sample_rate=16000, silence_threshold_ms=280):
        self.sample_rate = sample_rate
        self.silence_threshold_frames = int(silence_threshold_ms / 32)
        
        # Load lightweight Silero VAD model via torchscript
        self.model, _ = torch.hub.load(
            repo_or_dir='snakers4/silero-vad',
            model='silero_vad',
            force_reload=False,
            onnx=True
        )
        self.consecutive_silence_frames = 0
        self.is_speaking = False
        self.audio_buffer = bytearray()

    def process_pcm_frame(self, frame_bytes: bytes) -> dict:
        # Process a 32ms audio frame (512 samples at 16kHz mono).
        # Returns a dict indicating state transitions: speech_start, speech_end.
        self.audio_buffer.extend(frame_bytes)
        audio_int16 = np.frombuffer(frame_bytes, dtype=np.int16)
        audio_float32 = audio_int16.astype(np.float32) / 32768.0
        tensor_chunk = torch.from_numpy(audio_float32)

        # Get speech probability from neural model
        speech_prob = self.model(tensor_chunk, self.sample_rate).item()
        
        event = {"speaking": self.is_speaking, "turn_complete": False, "interrupted": False}

        if speech_prob > 0.55:
            self.consecutive_silence_frames = 0
            if not self.is_speaking:
                self.is_speaking = True
                event["speaking"] = True
                event["interrupted"] = True
                logger.info("Speech onset detected -> Triggering audio cancellation.")
        else:
            if self.is_speaking:
                self.consecutive_silence_frames += 1
                if self.consecutive_silence_frames >= self.silence_threshold_frames:
                    self.is_speaking = False
                    event["speaking"] = False
                    event["turn_complete"] = True
                    logger.info("Turn completion confirmed -> Dispatching accumulated buffer to STT.")
                    
        return event

4. Sub-40ms Barge-In Cancellation Handling

When the turn detector fires an interrupted event, the server must instantly suppress in-flight audio playback to allow the user to speak naturally:

async def handle_caller_barge_in(websocket_client, outbound_tts_task):
    # 1. Cancel in-flight TTS generation task immediately
    if outbound_tts_task and not outbound_tts_task.done():
        outbound_tts_task.cancel()
        
    # 2. Transmit immediate interruption frame to client WebRTC audio track
    await websocket_client.send_json({
        "type": "control.interrupt",
        "action": "clear_playback_buffer",
        "timestamp_ms": asyncio.get_event_loop().time() * 1000
    })

Key Architectural Takeaways

Achieving realistic, human-level voice interactions requires abandoning rigid, fixed-length silence timeouts. By implementing continuous frame-level neural voice activity detection (Silero VAD) paired with semantic utterance endpointing and instant buffer preemption, you reduce conversational latency to under 300ms while completely eliminating accidental interruptions.