Real-time Voice AI Agent

01The user speaks — what happens between that and the agent replying?

A streaming cascade end to end: VAD (voice activity detection — is there sound) gates streaming STT (speech-to-text), partial transcripts feed the LLM, and the LLM’s first tokens feed streaming TTS (text-to-speech) — every stage starts on the partial output of the previous, and no stage waits for a full result.

02How does the agent know the user is actually done talking?

A turn-detection ladder, not a silence timer: acoustic VAD hears sound in 10–50 ms, semantic endpointing judges whether the thought is finished — a pause mid-sentence is not a turn end — and only a completed turn triggers the reply.

03The user talks over the agent mid-reply?

Barge-in inside ~200 ms: detect the overlap, classify it — a real interruption stops the agent, a backchannel (a listener’s ‘mm-hm’) must NOT — then stop playback, flush the queued audio, cancel the in-flight LLM generation, and return to listening.

04The agent needs to look up an order mid-turn — does the line go silent?

Tool calls without dead air: the agent emits an acknowledgement phrase (‘let me pull that up’) the moment the tool starts, the tool runs while the phrase plays, and retrieval that cannot fit the voice budget moves off-turn.

05Calls arrive from real phones, not just the app?

Real phones are first-class: SIP/PSTN ingress (the public phone network’s signaling and audio) at 8 kHz narrowband, echo cancellation (AEC) so the agent’s own voice leaking back through the phone path never triggers VAD, and jitter buffers to absorb packet timing.

06The mobile network drops for two seconds mid-conversation?

One conversation = one stateful session pinned to a worker — audio in flight, partial transcripts, playback position, tool state — and a reconnect resumes the session from where the user left it, never restarts it.

01The user finishes a sentence — how long until they hear the agent?

Voice-to-voice under ~800 ms, target ~630 ms: VAD ~30 ms + STT final word ~150 ms + LLM first token ~300 ms + TTS first audio ~150 ms — achievable only because the stages overlap; above ~1–2 s the conversation collapses.

02The user talks over the agent — how fast must it shut up?

Barge-in end to end inside ~200 ms: detection, classification, playback stop, queue flush, and generation cancel all fit the budget — a human interrupted mid-word stops in about that time, and slower reads as being talked over.

03The agent was cut off mid-sentence — what does it think it said?

The transcript is what the user HEARD: on interruption the agent’s turn truncates at the playback position, never the generation position — conversation state and the user’s reality must not fork.

04A worker restarts or the network blips — does the caller start over?

Sticky and resumable: the session lives on one worker holding hot state, a network blip resumes audio position, partial transcript, and tool state within the same conversation — reconnect is invisible, restart is a product failure.

05A turn felt slow — which stage do you blame?

Every stage is observable per turn: timestamps for VAD trigger, STT final, LLM first token, and TTS first audio are recorded per utterance — the latency budget is enforced call by call, not averaged into invisibility.