Voice AI: The Caller Said Stop After the Tool Call

The caller says “stop.” The voice goes quiet immediately. A second later, the appointment still appears on the calendar. Nothing in that sequence is contradictory: playback stopped, but the external tool call had already crossed its commit boundary.

This is the voice AI failure that a latency demo rarely shows. A pleasant agent can produce its first syllable quickly, detect an interruption, and still leave behind an action the caller believes was cancelled. The defect is not simply “slow barge-in.” Four clocks were collapsed into one: incoming audio, turn admission, generated speech, and the side effect outside the voice stack.

A safer voice agent binds those clocks to a durable turn identity and keeps separate receipts for speech and tools. It can clear queued audio when the caller interrupts. It can cancel a reply that has not committed. It can refuse an unapproved write. What it cannot do is pretend that clearing a speaker buffer rewound an HTTP request that already reached a booking system.

Incident / “Stop” arrived between proposal and receipt

Consider a scheduling agent. The caller asks for Tuesday afternoon. The agent finds a slot, begins saying “I’ll book that now,” and dispatches a write to the calendar. Halfway through the sentence, the caller says, “Stop, make it Wednesday.” The audio system correctly detects new speech and clears the remaining synthesized response. The booking request, however, is already in flight.

If the application treats barge-in as a global cancellation flag, several bad outcomes become possible. It may report that nothing happened even though Tuesday was booked. It may retry after a timeout and create a second appointment. It may begin a Wednesday booking before learning the Tuesday result. Or it may transfer to a human with a transcript that shows an interruption but no authoritative record of the calendar write.

The fix is not a more dramatic “cancel everything” function. The system needs a narrow state machine:

turn admitted
  → tool proposed
  → caller approval bound
  → idempotency key bound
  → commit started
  → receipt known
  → caller informed

An interruption before commit started can discard the proposal. An interruption after that point changes the next operation from cancel to reconcile. That distinction should be visible in traces and durable state, not inferred later from a conversational transcript.

Transport / An audio stream is not a conversation state

Twilio Media Streams sends call audio over a WebSocket. Its documented messages include connection metadata, a stream identifier, a call identifier, monotonically ordered sequence numbers, media chunks, timestamps, stop events, and—in bidirectional streams—marks that can identify played or cleared outbound audio. These are valuable transport facts. They do not say whether the caller finished a thought or approved an external action.

The distinction matters during reconnects and retries. A media chunk belongs to a stream and has an order. A transcript result belongs to some portion of audio. A conversational turn belongs to an application-level identity. A tool proposal belongs to that turn, and its commit receipt belongs to an idempotency key. If any of those bindings are missing, the application should hold or block rather than assemble a plausible history from timestamps.

For outbound speech, Twilio documents a useful split between mark and clear. After the application sends media and a mark, the returned mark can indicate playback completion. If the application sends clear, Twilio empties buffered audio and returns marks for media that will not be played. That gives the voice layer evidence about what the caller could have heard. It still says nothing about whether a parallel tool request completed.

Store both receipts:

• Playback receipt: which generated audio was queued, played, or cleared.
• Tool receipt: which external action was proposed, approved, committed, rejected, timed out, or reconciled.

One should never stand in for the other. “The audio was cleared” is not evidence that an appointment was not created. “The tool returned success” is not evidence that the caller heard the confirmation.

Requirements / A pause is evidence, not permission

Voice activity detection answers a limited question: is speech present? LiveKit's Silero VAD documentation exposes the practical trade-offs through settings such as minimum speech duration, minimum silence duration, prefix padding, and activation threshold. A stricter threshold can reject soft speech; a looser threshold can interpret noise as speech. That is detector behavior, not proof of intent.

Endpointing adds another signal. Deepgram documents speech_final as a pause-based endpoint and lets applications configure the silence interval. Its utterance-end feature looks for a gap after the last finalized word. The documentation also warns about boundaries: utterance end is based on finalized word timings, does not detect every internal gap, and can fire while a person is continuing. A server event is convenient, but the application still owns the decision to admit a complete turn.

LiveKit's turn detector documentation makes the same point from another angle. VAD can hear silence without understanding that “Book the…” is unfinished. A semantic or context-aware detector can avoid some premature commits, but it introduces its own language support, timeout, fallback, and session behavior. When a realtime model already owns turn detection, enabling another competing detector can produce two sources of truth.

A robust admission policy therefore separates four states:

Tool authority must be bound after turn admission, not borrowed from a pause. “Yes” may approve a booking only when it is attached to the exact proposal, caller, session, target, and expiry. A final transcript containing “yes” is not a reusable permission token.

Failure modes / Stop speech, then decide what remains

Barge-in sounds like one feature, but the handler has at least three jobs. First, stop or clear outbound audio so the agent does not talk over the caller. Second, cancel any reply generation that has not crossed a useful commit point. Third, inspect tools associated with the interrupted turn.

That third branch is where designs diverge:

• A tool merely proposed can be discarded.
• A read-only tool already running can finish, but its result may be irrelevant to the new turn.
• An external mutation without bound approval and idempotency must not start.
• A mutation whose commit started but lacks a receipt is ambiguous and must be reconciled.
• A mutation with a known receipt remains part of the conversation history even if its spoken confirmation was cleared.

The interruption handler should be deliberately boring. It writes a new event against the same turn, clears playback, marks generation cancelled, and chooses one tool disposition. It does not delete prior events. It does not rewrite the transcript to make the interaction look cleaner. It does not send a compensating mutation until the original outcome is known.

Tool lane / Make the irreversible step small

A voice agent should prepare more than it commits. For a calendar booking, preparation might resolve the attendee, time zone, candidate slot, duration, calendar, and duplicate policy. The agent can read that proposal back in a short confirmation. Only the exact accepted proposal receives an approval record and an idempotency key.

The commit function should accept a narrow packet rather than the entire conversational state:

{
  turn_id,
  proposal_hash,
  principal,
  target,
  approval_receipt,
  idempotency_key,
  expires_at
}

This packet makes retries inspectable. If the network times out after dispatch, the agent queries by idempotency key or target receipt before retrying. If the caller interrupts, a new turn may alter the desired outcome, but it cannot silently reuse the old approval for a different time. If compensation is necessary, it becomes another explicit action—cancel appointment, refund charge, withdraw message—with its own authority and receipt.

Read tools need boundaries too, although the risk is different. An interrupted address lookup may finish harmlessly, but presenting its result in the next turn could be misleading. Tag every result with the turn and proposal it served. The reply composer should reject a stale result even when the data itself is correct.

Testing / Fifty-six races, zero phone calls

To test the state policy without manufacturing a provider benchmark, a deterministic local JavaScript fixture evaluated 56 declared event packets. The fixture covered call, stream, and turn binding; transport authentication; monotonic sequence order; duplicate and stale endpoints; final transcripts without final intent; output marks; barge-in; hangup; transfer; restart; read tools; external writes; approval; idempotency; and known or ambiguous receipts.

All 56 cases matched their declared result across 33 outcomes. Eight were ready or active, ten were held or kept listening, thirteen were blocked, four were dropped as stale or duplicate, seven required reconciliation, and ten cancelled or cleared output. Counts overlap where an outcome both cleared audio and entered reconciliation, which is exactly the point: interruption is not one terminal state.

Some cases establish precedence rather than happy paths. An unauthenticated duplicate is blocked as unauthenticated before duplicate handling. A non-monotonic event is blocked before it can be dropped as redundant. A restart without a durable turn receipt is held; if a tool had already started, the same restart enters reconciliation. Hangup cancels audio and generation, but a started tool with no receipt remains ambiguous.

The fixture hash is 4f3b4ee73a74e8e608506a811c5b0b3f8390ac0f251f07b0d23b20bac1d66077. The host had Node.js and FFmpeg available; SoX was absent. Those command checks were inventory only. The test captured or processed no audio and made zero microphone captures, phone calls, provider API calls, model calls, tool calls, external mutations, or credential uses.

This evidence supports the admission logic. It does not establish transcription accuracy, speech quality, acoustic interruption latency, carrier behavior, model quality, or vendor reliability. Those require controlled audio and provider runs.

Measurement / Retire the single latency number

“Voice latency” is too compressed to diagnose this class of incident. Measure the boundaries separately and preserve identifiers that let one session timeline be reconstructed:

• audio arrival to speech-start detection;
• last relevant audio to endpoint candidate;
• endpoint candidate to semantic turn admission;
• turn admission to first generated audio queued;
• queued audio to confirmed playback or clear mark;
• tool proposal to bound caller approval;
• tool dispatch to authoritative receipt;
• interruption detection to buffer clear;
• ambiguous commit to reconciliation result;
• transfer request to human ownership receipt.

LiveKit's observability documentation is useful here because it describes a session timeline that can correlate audio recordings, transcripts, traces, logs, tool calls, handoffs, and pipeline metrics. It also documents data-retention and recording controls. The practical lesson is broader than one platform: an operator needs a joined timeline, but recording audio and transcripts creates a privacy and retention obligation. Capture only what the incident policy can justify, and keep server-level crash or dispatch logs outside a session-only view.

Report distributions, not a single average. A fast median buffer-clear time can coexist with a dangerous tail of ambiguous tool commits. A quick first syllable can hide premature endpointing. The production objective is not “sub-second” in the abstract. It is a small, measured delay for each boundary plus a zero-tolerance policy for unbound writes.

Recovery / Restart from receipts, not from the transcript

When a voice worker restarts, the conversational transcript is helpful context but weak authority. It may omit cleared speech, partial recognition, a tool result that arrived after hangup, or a dispatch whose response was lost. Durable recovery starts from turn records, output marks, idempotency keys, and tool receipts.

For each incomplete turn, the recovering worker should ask:

1. Was the call and stream identity bound and authenticated?
2. Which media and transcript events were accepted in monotonic order?
3. Was the turn admitted, interrupted, transferred, or closed?
4. Which reply audio was queued, played, or cleared?
5. Was a tool proposed and approved for this exact proposal?
6. Did commit start, and is the receipt authoritative?
7. Does the caller need a correction, disclosure, transfer, or compensating action?

If the receipt is unknown, do not replay the write merely because the transcript ends with “yes.” Reconcile first. If the call has ended, the follow-up channel and consent policy must already be defined; the recovery worker should not improvise an SMS or another call.

Rollout / Make interruption boring before making it charming

Start with one narrow, read-heavy call type. Bind call, stream, turn, and sequence identifiers. Run one turn detector as the source of admission truth. Implement output marks and clear handling. Record tool proposals separately from commits. Require approval and idempotency for external writes. Add restart and hangup reconciliation before increasing autonomy.

The test set should include awkward speech, not only clean demos: mid-sentence pauses, backchannels, corrections, overlapping speech, background noise, a caller who interrupts confirmation, a late transcript final, duplicated WebSocket events, a tool timeout after dispatch, a worker restart, hangup during playback, and transfer while a receipt is pending. Use synthetic fixtures for state coverage and controlled audio for acoustic coverage. Neither replaces the other.

A human transfer is a terminal handoff, not an escape hatch. Bind the transfer target, include the current proposal and known receipts, and mark who owns the next action. The agent should not leave a calendar write unresolved while telling the caller that a person will handle it.

The mature voice agent is not the one that never hesitates. It is the one that can say, accurately, “I stopped speaking; the booking request had already started; I am checking whether it completed.” That sentence is less magical than an instant apology. It is also operationally honest.

Evidence boundary

This replacement is based on current official documentation for media transport, WebSocket events, turn detection, VAD, endpointing, utterance-end behavior, streaming transcription, and observability, plus one deterministic 56-case local state fixture. It includes no microphone capture, audio sample, phone call, provider account, provider API call, model call, TTS run, ASR quality test, tool invocation, customer data, credential, external mutation, acoustic latency measurement, or vendor comparison.

The article therefore makes no universal latency target and does not rank voice providers. It describes an application control boundary: speech interruption and external side-effect cancellation are different operations, and a started write requires reconciliation.

Primary sources

• Twilio Media Streams overview
• Twilio Media Streams WebSocket messages
• LiveKit turn detection and interruption management
• LiveKit Silero VAD
• LiveKit turn detector
• Deepgram endpointing
• Deepgram utterance end
• Google Cloud streaming speech recognition
• LiveKit agent observability