Semantic Intent Routing in Real-Time Voice Agents: Sub-15ms Local Vector Classification to Bypass Full LLM Inference

The Latency Budget of Human Conversation

In human spoken dialogue, the average turn-taking pause between speakers is approximately 200 to 300 milliseconds. When conversations exceed 800ms of latency, participants perceive noticeable lag, leading to awkward speech collisions, double-talking, and diminished trust. In a conventional cascaded Voice AI architecture (Speech-to-Text $\rightarrow$ LLM $\rightarrow$ Text-to-Speech), satisfying this latency budget is an extreme engineering challenge:

Even under optimal network conditions, passing every single user turn through a large language model makes sub-500ms conversational response times structurally impossible. Yet in production telephony and customer service voice streams, up to 45% of user utterances are deterministic conversational pivots: acknowledgments ("yes, that works", "uh-huh"), confirmations ("repeat that please", "say it again"), navigational triggers ("cancel my appointment", "speak to an agent"), or standard business FAQs.

1. Architecture of an Edge Semantic Intent Router

Rather than treating the LLM as the universal handler for every audio frame, high-performance voice gateways deploy a Semantic Intent Router operating directly inside the WebRTC media worker. The moment speech-to-text emits a finalized utterance transcript, the router evaluates whether the query maps to a known deterministic state with a calibrated confidence threshold.

[WebRTC Audio Input] ──> [Streaming STT Transcript]
                                │
                                ▼
                 [Sub-15ms Semantic Intent Router]
                                │
             ┌──────────────────┴──────────────────┐
             ▼                                     ▼
     (High Confidence Match)              (Uncertain / Complex)
             │                                     │
   [Pre-Cached Opus Audio]              [Streaming LLM Generation]
             │                                     │
             ▼                                     ▼
   [Instant WebRTC Playback]            [Dynamic TTS Vocoder]
      (Total: 180ms Latency)             (Total: 650ms Latency)

2. Running Quantized Bi-Encoders with ONNX Runtime

To classify utterances within a strict 15ms budget on standard CPU gateway hardware, the router utilizes a quantized bi-encoder model (such as bge-small-en-v1.5 or all-MiniLM-L6-v2) compiled to an 8-bit quantized ONNX graph. Unlike heavy cross-encoders, bi-encoders map user input to a dense embedding space in a single forward pass:

import numpy as np
import onnxruntime as ort
from transformers import AutoTokenizer

class FastIntentClassifier:
    def __init__(self, model_path: str, tokenizer_name: str):
        # Configure ONNX Runtime for multi-threaded low-latency CPU execution
        opts = ort.SessionOptions()
        opts.intra_op_num_threads = 2
        opts.execution_mode = ort.ExecutionMode.ORT_SEQUENTIAL
        opts.graph_optimization_level = ort.GraphOptimizationLevel.ORT_ENABLE_ALL

        self.session = ort.InferenceSession(model_path, opts, providers=['CPUExecutionProvider'])
        self.tokenizer = AutoTokenizer.from_pretrained(tokenizer_name)
        self.intent_vectors = {}

    def embed_utterance(self, text: str) -> np.ndarray:
        inputs = self.tokenizer(
            text, padding=True, truncation=True, max_length=32, return_tensors="np"
        )
        ort_inputs = {
            'input_ids': inputs['input_ids'].astype(np.int64),
            'attention_mask': inputs['attention_mask'].astype(np.int64)
        }
        outputs = self.session.run(None, ort_inputs)
        # Apply mean pooling over token embeddings
        token_embeddings = outputs[0]
        mask = np.expand_dims(inputs['attention_mask'], -1)
        embedding = np.sum(token_embeddings * mask, axis=1) / np.clip(mask.sum(axis=1), a_min=1e-9, a_max=None)
        # L2 Normalization for fast cosine distance via dot product
        norm = np.linalg.norm(embedding, axis=1, keepdims=True)
        return (embedding / norm).squeeze(0)

3. Pre-Cached Audio Buffers: Instantaneous Playback

When an incoming utterance matches an intent (such as repeat_last_phrase with $ ext{cosine\_similarity} > 0.86$), the voice gateway completely bypasses both the LLM and the Text-to-Speech synthesis engine. Instead, it streams pre-encoded 20ms Opus audio packets directly from an in-memory ring buffer into the user's WebRTC audio track.

The resulting perceived latency drops from ~700ms down to ~180ms (the bare minimum required for STT endpointing and RTP transmission). The user experiences an instantaneous, natural human reaction, while the server conserves precious GPU inference resources for generative turns that genuinely require complex reasoning.