Voice AI Latency Budget: Why 500ms Is the Line

Voice AI starts feeling broken around 500 milliseconds of round-trip silence, because that's roughly the point where a listener's brain stops reading the pause as "the other party is thinking" and starts reading it as "the system stalled." Human conversation runs on turn-taking gaps of about 200ms; a voice agent gets a little more slack than that, but not much — and six separate stages between a spoken word and a spoken reply spend that slack fast.

Quick answer: Conversational voice AI runs on a fixed, shared latency budget, not an elastic one. Human turn-taking research puts natural response gaps around 200-300ms; once total round-trip latency crosses roughly 500ms, users stop feeling heard and start feeling processed. Capture, endpointing, transcription, inference, synthesis, and playback all draw from that same shrinking pool.

The Human Benchmark: What Counts as "Normal" in a Conversation

The real spec for voice AI latency isn't an engineering convenience number — it's a human one. Conversation-analysis researchers have measured how long people actually wait before responding in real dialogue, and the answer clusters tightly around 200ms, whether the language is English, Japanese, or Danish.

That finding comes from a landmark study by Tanya Stivers and colleagues at the Max Planck Institute for Psycholinguistics, published in PNAS in 2009, which timed turn-taking gaps across ten unrelated languages. The average gap was close to 200ms, with cross-linguistic variation from near-simultaneous overlap in some languages to roughly 470ms in others. The point isn't the exact number — it's that humans have a shared, deeply ingrained sense of "normal" response timing, and it's fast.

Telecom engineers arrived at a similar boundary from a completely different direction. The ITU-T G.114 recommendation, the standard used to design international phone networks, treats one-way delay under 150ms as imperceptible, 150-400ms as acceptable but degrading, and anything past 400ms as unsuitable for most interactive use. Round-trip equivalents for "acceptable" voice calls land in roughly the 300-800ms range — a ceiling the phone network has enforced for decades without most callers ever noticing it exists.

UX research generalizes the same shape without being voice-specific. Jakob Nielsen's classic response-time thresholds — popularized through Nielsen Norman Group's usability research — hold that 0.1 second feels instantaneous, 1.0 second keeps a user's flow of thought uninterrupted, and 10 seconds is the outer limit before attention is lost entirely. Voice compresses that middle threshold hard: a synchronous spoken channel makes a one-second gap read as a stall, not a "still with you" pause the way a webpage spinner might.

Real products have started converging on the same number. When OpenAI launched GPT-4o's voice mode in May 2024, it marketed an average audio response latency of 320ms and explicitly compared it to human conversational response time. That's not a coincidence — it's a team optimizing toward the exact benchmark this section describes.

Benchmark	Source	Threshold	What it implies for voice AI
Natural turn-taking gap	Stivers et al., 2009, Max Planck Institute for Psycholinguistics (PNAS)	~200ms average, ~0-470ms range across languages	This is the target, not a stretch goal
Telephony one-way delay	ITU-T G.114	<150ms imperceptible; 150-400ms acceptable; >400ms degraded	Round-trip "acceptable" calls land near 300-800ms
Response-time perception	Jakob Nielsen / Nielsen Norman Group	0.1s instant; 1.0s uninterrupted flow; 10s attention limit	Voice compresses the 1-second grace period because it's synchronous
Shipped agent benchmark	OpenAI, GPT-4o voice launch, May 2024	~320ms average audio response, marketed as human-comparable	Confirms ~300-500ms is where a voice agent starts to "sound human"

The Six Stages That Eat Your Latency Budget

A voice round trip isn't one system — it's six handoffs, each adding delay before the user hears anything back: capture, endpointing, transcription, model inference, speech synthesis, and playback. Add them up sequentially and you blow past 500ms almost by accident; the only way back under budget is to stop treating them as sequential.

Each stage has a typical range, a job to do, and a way teams accidentally overspend it:

Stage	Typical range	What's happening	Where teams lose time
Capture	20-50ms	Mic buffers audio frames before anything can process them	Oversized buffers set "for safety"
Endpointing (`VAD`)	150-500ms	Deciding the user has actually stopped talking	Conservative silence timeouts to avoid cutting people off
Network transport	40-150ms round trip	Encode, upload, TLS handshake, download, if server-based	Wrong server region; TCP instead of a real-time transport
`STT` finalization	100-300ms	Producing a final transcript after the endpoint fires	Waiting for a "final" instead of acting on interim partials
`LLM` time-to-first-token (`TTFT`)	150-500ms+	Processing context and deciding the first words of a reply	Long system prompts and full-turn generation before speaking
`TTS` time-to-first-byte (`TTFB`)	100-300ms	Synthesizing the first audio chunk	Waiting for a whole sentence or paragraph before synthesizing
Playback	20-60ms	Jitter buffer and audio decode on the way out	Buffers sized to hide network jank instead of fixing it

Stack those stages naively — wait for each one to fully finish before starting the next — and you land somewhere around a full second, well past where the conversational illusion holds:

Capture         [█░░░░░░░░░░░░░░░░░░░]   30ms
Endpointing     [██████████░░░░░░░░░░]  300ms   <- usually the single biggest cost
Network         [███░░░░░░░░░░░░░░░░░]   80ms
STT final       [██████░░░░░░░░░░░░░░]  180ms
LLM TTFT        [████████░░░░░░░░░░░░]  250ms
TTS TTFB        [█████░░░░░░░░░░░░░░░]  150ms
Playback        [██░░░░░░░░░░░░░░░░░░]   40ms
                                        --------
Naive sequential total:                1,030ms

Now overlap the same stages — start synthesizing before the model finishes, start the model before the transcript is fully final — and the perceived total drops close to the human-comparable range, without changing a single vendor:

Capture + endpointing (unavoidable floor)      330ms
Everything else, streamed and overlapped       ~150-200ms
                                                --------
Streamed pipeline total:                       ~480-530ms

That gap between 1,030ms and ~500ms is the entire argument of this article: latency isn't primarily a compute problem, it's an architecture problem.

Endpointing: The Tax Nobody Puts on the Roadmap

Endpointing — deciding the user has actually finished talking, not just paused to think — is usually the single largest chunk of the whole latency budget. It's often 150-500ms of pure waiting, and it barely shows up in demos, because scripted demos rarely include a mid-sentence "um."

The mechanics are simple and the tradeoff is genuinely hard, not a solvable bug. A VAD (voice activity detection) system has to wait out a silence timeout before declaring "the user is done." Set that timeout too short and you cut people off mid-thought — a much worse experience than a slow reply. Set it too long and every single reply feels sluggish, even when transcription and inference are instant.

A few real mitigations, roughly in order of maturity:

Semantic endpointing — using the partial transcript itself to judge whether the utterance is grammatically or pragmatically complete, rather than relying purely on acoustic silence.
Adaptive timeouts — shortening or lengthening the wait based on a speaker's own cadence, so a fast talker isn't held to a slow talker's silence threshold.
Explicit turn signals — push-to-talk, a wake word, or a visible "I'm listening" state that removes the ambiguity entirely, at the cost of some naturalness.
Barge-in support — letting the user interrupt the agent mid-reply rather than trying to perfect the timeout in the other direction; this doesn't reduce endpointing latency, but it makes the cost of a slightly-too-long timeout much less painful.

None of these make endpointing free. What they do is convert a fixed acoustic tax into a tunable one, which is the most any team should expect to achieve here.

The Streaming Trick: Start Talking Before You're Done Thinking

The single biggest latency win available to most teams is architectural, not computational: stream every stage instead of waiting for it to finish, so the first word of the model's answer reaches the speech synthesizer while the fortieth word is still being generated. Done well, this turns a sequential waterfall into an overlapping relay and can cut perceived latency by half without touching model choice.

In practice, this means three things happening at once instead of in sequence:

Stream the model output. Don't wait for a complete response — chunk generation by clause or sentence boundary and forward each chunk to synthesis the moment it's ready, rather than after the full turn is drafted.
Stream the synthesis. Use a TTS engine built for incremental synthesis, where TTFB matters more than total render time, so audio starts before the whole sentence is even fully written.
Act on interim transcripts. Interim STT results can kick off speculative work — retrieving context, shaping a draft response — before the final transcript locks in, as long as the system can gracefully revise if the final transcript changes meaning.

This is exactly why open frameworks built specifically for real-time voice agents — Pipecat (Daily's open-source voice agent framework) and LiveKit Agents among them — pipeline audio, transcript, and token streams stage-to-stage instead of batching. The architecture is the optimization; the individual model or TTS vendor is comparatively interchangeable.

Streaming isn't free of tradeoffs, though. Speaking a sentence before the full context is resolved means occasionally starting a reply that later context invalidates — a "wait, let me rephrase that" moment a designer has to plan for explicitly, not discover in production.

Prompt structure matters here too: a leaner, more deliberately organized system prompt reduces the context the model has to process before emitting its first token. That's one of several reasons prompt construction choices have a direct, measurable effect on TTFT — a topic our guide to prompt design covers in more depth.

Model selection carries its own latency-cost tradeoff worth naming honestly: bigger, more capable models are frequently slower and more expensive per token, so choosing model size for a voice pipeline is really a three-way negotiation between latency, cost, and quality — the kind of tradeoff our guide to AI economics walks through in more depth.

Building Your Own Latency Budget

Treat the ~500ms ceiling like any other fixed resource budget — a page's performance budget for load time, say — and allocate it deliberately across stages rather than hoping the total comes in under budget by accident. Instrument every stage independently, give each one a target, and treat any stage that blows its allocation as a bug, not a rounding error.

A practical sequence for doing this:

Instrument each of the six stages separately, with real timestamps at stage boundaries — not just one end-to-end number that hides which stage is actually slow.
Assign every stage a ceiling that sums to your target total, and revisit the allocation whenever you swap a vendor, not just when users complain.
Design a deliberate degraded mode for when a stage blows its budget — a short filler cue, a cached "let me check on that," anything better than dead air.
Log latency distributions, not averages. A single slow STT call in twenty will sink a live demo even if the p50 looks great; p95 and p99 are the numbers that actually predict user-visible failures. Treating this kind of stage-level telemetry as a real data asset — not an afterthought — is the same discipline covered in our guide to AI data strategy.
Plan explicitly for the interruption case. Filler audio and backchannel cues ("mm-hmm") that mask latency can accidentally talk over a genuine user interruption if barge-in detection isn't built in from the start — a trust question as much as a technical one, and one worth reading alongside our guide to AI safety before you ship it.

None of this is exotic. It's the same performance-budget discipline web teams have used for page load for a decade, applied to a channel where the tolerance is roughly ten times tighter. For the wider set of design decisions across voice, video, and multimodal interfaces beyond just latency, our complete guide to multimodal voice interfaces covers the rest of that territory.

What On-Device Capture Gets Right (and Why It Isn't the Whole Story)

Prodinja's journal and note-taking entry points use the browser's built-in Web Speech API for dictation rather than streaming audio to a server for transcription, which is a clean, honest illustration of the capture-stage lesson above: skipping a network round-trip for the very first stage of the pipeline keeps perceived latency low without needing a server-side voice pipeline at all.

It's worth being precise about what that does and doesn't solve. On-device recognition removes capture-stage network latency and gets local STT results back essentially instantly, which is exactly right for a dictation use case — typing by voice into a journal entry or a field note.

It does not touch the other five stages of a full conversational round trip, because dictation has no inference, synthesis, or playback stage to begin with; there's no spoken reply to generate. For teams weighing when voice should be a capture feature versus a full back-and-forth agent, our piece on voice dictation as a capture feature goes further into that distinction.

The broader lesson still holds regardless of which side of that line a product sits on: every network hop removed from any stage is milliseconds a team doesn't have to claw back somewhere else in an already-tight budget. Whether that's on-device capture for dictation or overlapped streaming for a full conversational agent, the arithmetic is the same — the pool is fixed, and every stage is spending from it.

Key Takeaways

Voice AI's real latency spec comes from human turn-taking research, not engineering convenience — natural conversational gaps average roughly 200ms, per Stivers et al.'s cross-linguistic study.
Total round-trip latency past roughly 500ms is where users stop feeling heard and start feeling like they're talking to a system that's stalled.
A voice round trip has six stages — capture, endpointing, STT, LLM inference, TTS, playback — and adding them sequentially routinely produces latencies well over one second.
Endpointing (deciding a user has actually finished speaking) is usually the single largest, most overlooked cost in the whole budget, often 150-500ms on its own.
Streaming every stage — model tokens into synthesis, interim transcripts into speculative work — is the highest-leverage fix available, often cutting perceived latency in half without a faster model.
Treat latency like a fixed resource budget: instrument each stage, set per-stage ceilings, and monitor p95/p99, not averages.
Removing a network hop from even one stage, like on-device speech capture, is a real and honest way to buy back milliseconds elsewhere in the budget.

Frequently Asked Questions

What is an acceptable latency for voice AI?

Most teams should target total round-trip latency under roughly 500ms, with anything approaching the 200-300ms human turn-taking baseline feeling genuinely natural. Beyond about 800ms-1,000ms, users generally register the system as slow or unresponsive rather than merely "thinking," based on the same tolerance ranges telecom standards like ITU-T G.114 have long used for voice calls.

Why does voice AI feel slower than texting with a chatbot?

Because a spoken channel is synchronous and humans have an unconscious, deeply calibrated sense of normal turn-taking timing, while a text chat has no equivalent real-time expectation. A one-second delay in a chat thread barely registers; the same delay in a live voice exchange reads as a stall, because there's no visual "typing" cue filling the silence the way there is in text.

How do you reduce time-to-first-token in a voice pipeline?

Shorten and simplify what the model has to process before it can start responding — a leaner system prompt, less unnecessary context, and streaming generation so the first token can be spoken while the rest of the reply is still being produced. Model choice matters too: smaller or more latency-optimized models generally post lower TTFT than larger general-purpose ones, at some quality cost.

What is endpointing in speech recognition?

Endpointing is the process of deciding, in real time, that a user has actually finished their turn rather than just paused mid-thought. It typically relies on a silence timeout from a VAD system, sometimes combined with semantic cues from the partial transcript, and it's frequently the single largest source of avoidable latency in a voice pipeline.

Does a bigger, smarter model always mean worse latency?

Generally yes, directionally — larger models tend to have higher per-token latency and cost, though the gap narrows with model-specific optimization and hardware. That makes model selection for a voice agent a genuine three-way tradeoff between latency, cost, and response quality, rather than a simple "biggest model wins" decision.