VAMSHI.

Agentic Engineering

Voice Agents Need a Turn-Taking Architecture

Reliable voice agents coordinate streaming audio, dialogue state, interruption, tool latency, confirmation, recovery, and conversational evaluation.

5 min read
voice agents realtime ai conversational ai agent architecture multimodal ai
A duplex timeline shows user speech, partial understanding, agent response, user interruption, tool work, stale results, and recovery.

A text agent can pause for eight seconds and still appear thoughtful.

A voice agent that goes silent for eight seconds feels broken.

Voice turns agent design into a timing problem. The system must decide when a person has finished, whether to begin responding, how to stop when interrupted, what to do while a tool is slow, and how to repair a misunderstanding without restarting the conversation.

The product is not text with speech attached. It is a turn-taking system.

Choose the conversation architecture first

There are two common designs.

ArchitectureFlowBest when
Speech to speechLive audio enters and leaves one realtime sessionNatural conversation, low first-audio latency, interruption, expressive audio
Chained pipelineSpeech recognition → text agent → speech synthesisIntermediate text control, reuse of an existing text workflow, easier component inspection

OpenAI’s current voice-agent guide describes the same tradeoff. Speech-to-speech reduces boundaries and supports natural barge-in. A chained pipeline gives the application more explicit control over transcript, reasoning, and speech components.

Neither choice removes the need for business state, tool policy, confirmation, and recovery.

Budget latency by stage

End-to-end latency tells you the conversation is slow. It does not tell you why.

Measure:

speech detected
→ partial understanding available
→ end of turn detected
→ response decision made
→ first audio emitted
→ tool result available
→ final response complete

Track p50 and p95. Users remember the worst pauses, especially after simple questions.

Some delay is unavoidable. The product needs behavior for it. A short acknowledgment can help while checking an appointment system. Repetitive filler is annoying, and claiming “I’ve booked that” before the tool confirms it is dangerous.

The agent should distinguish conversational progress from action completion.

Interruption is a state transition

Suppose the user says:

Move my appointment to Friday afternoon.

The agent begins searching. Before it responds, the user says:

Actually, make it Monday morning at the downtown office.

The system must:

  1. stop or duck the current audio quickly
  2. capture the corrected date, time, and location
  3. invalidate the Friday search if possible
  4. mark any late Friday result as stale
  5. run the Monday search
  6. confirm the final details before booking

Stopping speech is not the same as canceling work. A tool call may already have created a side effect. Every consequential call needs an identity and state so the system can tell whether it is pending, completed, canceled, or stale.

Late results should never hijack the current turn.

Keep dialogue state outside the transcript

Conversation history is useful context. It is a weak state store.

Represent the workflow explicitly:

intent: reschedule appointment
date: Monday [confirmed]
time: morning [confirmed]
location: downtown [confirmed]
selected_slot: none
booking_status: not submitted
pending_tool: availability_search_42

Now interruption, correction, and recovery have something stable to update.

The transcript may contain three different dates. The state object says which one the system currently believes and why.

Confirmation should follow consequence

Do not confirm every harmless detail; the conversation becomes robotic. Do confirm information immediately before an action that is difficult to reverse or costly to get wrong.

For an appointment, that may be date, time, location, and cancellation consequence. For a payment, it includes amount, recipient, and funding source. For support, it may include the account or record being changed.

The voice should make the boundary clear:

  • “I found two openings” is evidence.
  • “You selected Monday at 9:30” is current state.
  • “The appointment is booked” is a completed side effect.

Those sentences should not be interchangeable.

Conversational alignment is procedural

Tone matters, but alignment is more than sounding friendly.

A useful voice agent:

  • yields when interrupted
  • does not repeat sensitive details unnecessarily
  • distinguishes uncertainty from completion
  • remembers confirmed constraints
  • repairs a misunderstanding briefly
  • avoids filler when it has no new information
  • hands off when the request exceeds its authority

A pleasant voice cannot compensate for an incorrect commitment.

Evaluate the timeline, not only the transcript

Two conversations can produce the same text and feel completely different.

Test scenarios with:

  • interruption during agent speech
  • correction mid-utterance
  • hesitation and long pauses
  • background speech
  • accent and code-switching
  • overlapping speakers
  • slow tools
  • late tool failure
  • stale tool results
  • failed side-effect cancellation

Score task success, factual accuracy, first-audio latency, interruption latency, confirmation behavior, repair quality, unnecessary words, and user effort.

Review the audio, event trace, dialogue state, and tool timeline together. A transcript cannot show that the agent kept talking for two seconds after the user tried to interrupt.

The architecture of a good voice agent is visible in the moment the user changes their mind—and the system changes with them.

Further reading