Turn-Taking and State: Why Voice Agents Forget What You Just Said

A voice agent "forgets" what you said because it never actually held onto it — most systems treat each turn as a fresh transcript, not a state update. The fix isn't a bigger model; it's three mechanics working together: endpointing (knowing a turn ended), a dialog-state object that persists across turns, and reference resolution that maps "the second one" back to something concrete.

Quick Answer: Voice agents lose context because turn-taking and state are usually bolted on as an afterthought to the ASR/LLM pipeline. Spec explicit endpointing thresholds, a persistent dialog-state object, and a reference-resolution step — naturalness comes from timing and memory management, not model quality alone.

Why Voice Agents Seem to "Forget"

The forgetting isn't a memory failure in the LLM sense — it's an architecture gap between turns. Most voice pipelines run automatic speech recognition (ASR), pass a transcript to an LLM, get a response, and synthesize speech, treating each round as close to stateless unless someone deliberately wires up persistence. When that wiring is thin, the second turn loses what the first turn established.

This matters because conversational UX research — going back to conversation analysis work by Harvey Sacks, Emanuel Schegloff, and Gail Jefferson on turn-taking in the 1970s — showed that human dialog runs on precise, shared timing and referential shortcuts, not just semantic content. Voice agents inherit the referential shortcuts (users say "that one," "the second option," "cancel it") without inheriting the machinery that makes those shortcuts resolvable.

Three failure points compound:

1. Endpointing decides a turn is "done" before the user finished a thought, so the next turn starts against incomplete input.
2. Dialog state doesn't persist the right fields, so "it" and "the second one" have nothing to bind to.
3. The LLM context window resets or truncates, dropping slot values a rule-based system would have kept indefinitely.

Any one of these alone produces the "forgot what I said" complaint. In practice they usually stack, which is why the fix has to be structural, not a single prompt tweak. For the broader landscape this sits inside, see the multimodal voice complete guide.

Endpointing: The Pause-Length Problem

Endpointing is the system's decision that a user has finished speaking and it's the agent's turn — get the threshold wrong and you either cut people off mid-thought or leave dead air that feels broken. There is no single correct pause length; the right value depends on utterance type, and a fixed global timeout is the most common root cause of "the agent forgets" complaints.

The tension is symmetric. Cut a threshold too short and the system barges in on a user still formulating a number, a name, or a qualifying clause — and then has to silently discard whatever context that half-utterance carried. Set it too long and users perceive the system as slow or unresponsive, which independently degrades trust in a very similar way to actual state loss.

Consumer devices tune this asymmetrically by content type, and Google's own Assistant and Amazon's Alexa documentation on end-of-speech detection both describe adaptive, not fixed, silence windows. A few practical anchors:

Adaptive endpointing beats a fixed timeout. Systems that adjust the silence window based on utterance context (a trailing preposition, a rising intonation, a number sequence in progress) recover a meaningful share of premature cutoffs without materially increasing perceived latency. This is a design decision to spec explicitly, not leave to a vendor's default — see the related discussion in the voice latency budget piece on how endpointing timing eats into your overall response budget.

Barge-In as a Release Valve

Barge-in — letting a user interrupt the agent mid-response — is a partial mitigation for endpointing errors, not a substitute for tuning them. If a user can always cut in and redirect, an occasionally-too-long pause threshold is more forgivable. Spec barge-in as a first-class requirement, not a nice-to-have, because it changes how aggressively you can tune endpointing elsewhere.

Dialog State: What Actually Needs to Persist

Dialog state is the structured record of what's been established in a conversation so far — not the raw transcript, but the resolved facts, open slots, and pending confirmations a system needs to interpret the next turn correctly. PMs underspec this because it's invisible in a demo that only exercises single-turn exchanges.

A workable state object borrows from the slot-filling frame used in traditional dialog systems research (the classic frame-based approach documented in Jurafsky and Martin's Speech and Language Processing, still the standard NLP/dialog-systems reference) and adds a short-term reference buffer for spoken deixis. At minimum, track:

• Intent stack — the current goal, plus any suspended goals if the user detoured ("actually, before that...")
• Slot values — named parameters already collected (date, quantity, product, recipient) with a confidence score per slot
• Reference buffer — the last N mentioned entities, ordered by recency, for resolving "it," "that," "the second one"
• Confirmation state — what the system has and hasn't yet verified back to the user
• Turn history metadata — not full transcript, just enough to detect topic shifts or repetition

A Simple State Model PMs Can Spec Against

You don't need a research-grade dialog manager to fix most "forgetting" bugs. A minimal state machine with four regions covers the majority of real multi-turn voice flows:

Spec this as a literal schema in your PRD, with explicit rules for when each field gets written, read, and cleared. "Clear reference_buffer when active_intent changes" is the kind of rule that prevents an agent from resolving "the second one" against an entity list from two topics ago — a bug that's nearly invisible until someone actually tests topic switches.

Reference Resolution: Making "The Second One" Work

Reference resolution is the step that maps a pronoun or deictic phrase to a concrete entity from recent context — without it, "the second one" is meaningless to the system even if the human speaking it knows exactly what they mean. This is the single most common trigger for the "it forgot" complaint, because it's the most human-feeling failure: a person would never lose track of "the second one" seconds after saying it.

The mechanics are simpler than they sound if the reference buffer above is properly maintained:

1. List responses populate the buffer in order. When the agent reads back three options, each gets an index and an entity reference pushed onto the buffer.
2. Ordinal and demonstrative phrases resolve against buffer position, not against a fresh re-parse of the whole conversation. "The second one" looks up index 2; "that one" looks up the most recent single mention.
3. Ambiguity gets a clarifying turn, not a guess. If the buffer has two equally recent candidates for "it," the system should ask rather than pick — a wrong guess erodes trust faster than an extra confirmation turn.
4. The buffer has a short half-life. Entities older than roughly one or two intent-switches should expire, or you get the opposite bug: resolving "it" against something from three topics ago.

Anaphora resolution has decades of computational linguistics behind it (Winograd Schema-style challenges specifically test this), but production voice systems don't need academic-grade coreference — they need the buffer above, maintained consistently, and a conservative disambiguation policy. Get error handling right here and it compounds with recovery patterns covered in error correction for multimodal interfaces.

Modeling the Loops Before You Build

A multi-turn dialog is, structurally, a set of feedback loops: user input updates state, state changes what the agent says, what the agent says shapes the next input, and confirmation loops either close cleanly or spiral into repeated clarification. Most teams build this straight into code and only discover the loop that never terminates — the confirmation that keeps re-triggering itself — once real users hit it.

Sketching the state transitions as a causal-loop diagram before writing a single prompt makes reinforcing loops (a misheard slot value that triggers a re-ask, which gets misheard again) visible on paper instead of in a support ticket. Prodinja's Systems Engineering tool is built for exactly this kind of diagramming — it lets you lay out entities and transitions as a causal-loop diagram and runs real feedback-loop detection over the graph, so a runaway confirmation loop or an unreachable state shows up as a structural flaw in the model before it becomes a production bug. It's a genuinely useful pre-build step for exactly the class of state-machine problem this article describes, not a replacement for testing the built system.

Common Mistakes When Speccing Multi-Turn Voice

Most gaps trace back to specs that describe individual turns but never describe the transitions between them. A few patterns show up repeatedly across teams new to this:

• Specifying happy-path only. A PRD that shows one clean three-turn exchange rarely specs what happens when the user changes their mind mid-flow, repeats themselves, or answers a different question than asked.
• Treating the LLM context window as the dialog state. Raw conversation history is not the same as structured state — a long transcript can still fail to answer "what slot values are confirmed right now?"
• No explicit timeout/retry policy. What happens after two failed clarification attempts is a product decision, not an implementation detail to leave to the model.
• Ignoring topic-switch detection. Users detour ("actually, what's my balance first") constantly; state that doesn't handle a suspended intent stack breaks on this immediately.
• No confidence thresholds per slot. Low-confidence ASR output silently written into a slot as if it were certain is a frequent, quiet source of "the agent got it wrong."

This is also where connecting voice to other capture modes pays off — see voice dictation as a capture feature for how the same state-and-correction thinking applies when voice is one input among several, and customer journey mapping for placing multi-turn voice moments inside a broader emotional arc rather than specifying them in isolation.

Key Takeaways

• Endpointing thresholds should vary by utterance type — short commands, open queries, numeric input, and emotionally loaded speech each tolerate different pause lengths; a single fixed timeout is the most common root cause of premature cutoffs.
• Dialog state is a structured object, not a transcript — track intent stack, slot values with confidence scores, a reference buffer, and confirmation status explicitly, and spec when each field is written, read, and cleared.
• Reference resolution needs an ordered, short-lived buffer of recently mentioned entities so "the second one" or "that" resolves correctly, with a conservative policy that asks rather than guesses under ambiguity.
• Barge-in is a release valve for endpointing errors, not a substitute for tuning pause thresholds — spec it as a first-class requirement.
• Model the dialog as feedback loops before building it — a causal-loop diagram surfaces runaway confirmation loops and dead-end states on paper, which is far cheaper than finding them in production.
• Most "forgetting" bugs trace to unspecified transitions, not model weakness — happy-path-only specs, missing topic-switch handling, and absent confidence thresholds are the recurring gaps.