AI Agents & RAG

Verified against LiveKit Agents · 2026-08-05

Design turn-taking for a voice agent before latency and interruptions make it feel broken

A prompt for specifying exactly when a real-time voice agent speaks, stops, and resumes — silence thresholds, backchannel filtering, low-confidence transcript handling — decided as concrete numbers tied to a latency budget, before the conversational logic gets implemented and feels robotic in ways nobody planned for.

LiveKit AgentsDeepgramElevenLabsOpenAI Realtime API5 fillable variables

The prompt

Ready to copy — highlighted parts are example details you can swap.

You are designing the turn-taking and interruption behavior for a real-time voice agent, before the conversational logic gets implemented against LiveKit Agents with Deepgram STT and ElevenLabs TTS. Your job is to specify exact rules for when the agent speaks, stops, and resumes — not to describe voice-agent best practices in the abstract.

CONVERSATION CONTEXT
an appointment-scheduling agent for a dental office, calm and patient tone, callers are often older and speak more slowly than average

LATENCY BUDGET
under 800ms from end-of-speech detection to the start of the agent's audio response — every rule below has to be judged against this number, since a turn-taking rule that sounds natural in a text transcript can feel broken in real time if it adds latency nobody accounted for.

TURN-TAKING RULES
- Define the silence threshold that signals the user has finished speaking — not too short, cutting off a user mid-thought who paused to think, and not too long, an awkward dead-air gap before the agent responds. State the actual number in milliseconds and what evidence supports it for this use case specifically, not a generic "a second or so."
- Define how the agent signals it's still processing when generation takes longer than the silence threshold would suggest a response is imminent — a filler phrase, a soft acknowledgment sound, or explicit silence handling. Silence with no signal is the single most common way a voice agent reads as broken, even when it's about to respond correctly.

INTERRUPTION HANDLING
- State what happens when the user starts speaking while the agent is mid-response: does the agent stop immediately, finish its current sentence and then stop, or ignore short interjections under under 400ms of user audio, and matching a short backchannel word list (mhm, yeah, right, okay) as backchanneling rather than a real interruption?
- State what the agent does with the sentence it was cut off mid-way through — silently continue if the interruption turns out to be a backchannel, discard the rest and treat it as a full turn change, or explicitly acknowledge the cutoff before deciding? Pick one and justify it against the use case — a support call and a casual voice assistant warrant different answers.

FAILURE RECOVERY
- What happens when speech-to-text returns a low-confidence or garbled transcript — does the agent ask the user to repeat themselves immediately, or attempt to respond to its best guess and let the conversation self-correct? State the confidence threshold that decides which path, tied to Deepgram's per-utterance confidence score; below 0.6 triggers a clarifying repeat-back rather than a guessed response.
- What happens on a long silence after the agent asks a question — a fixed timeout before the agent checks in, stated as a number, not "after a while."

OUTPUT FORMAT
A state-by-state description: for each conversational state — agent speaking, user speaking, both silent, ambiguous — the exact rule for what triggers a transition to another state, with every threshold as a concrete number tied to the latency budget.

Customize

Optional — swap in your own details for the highlighted parts above.

Why this works

Tying the silence threshold to the specific caller population — older callers who speak more slowly and pause more between thoughts — rather than using a generic default tuned on average conversational speech, matters because a threshold that works well for the population it was implicitly tuned on will systematically misfire on a population with different pacing: a caller pausing to recall an appointment date will get cut off mid-thought by a threshold set for faster speakers, and every cutoff reads to that caller as the system not listening properly, which is a much worse experience than an extra 200ms of dead air would have been. There is no single correct silence threshold in the abstract — it's a number that has to be chosen against the actual population using this specific agent. Requiring an explicit filler or acknowledgment signal during processing delay addresses a specific and well-documented property of how humans interpret silence in real-time conversation: silence past a very short window gets interpreted as 'not heard' or 'connection dropped,' not as 'thinking,' because that's how human-to-human conversation actually behaves. A voice agent that goes fully silent while a slower generation call completes will read as broken even in the common case where it's about to produce a perfectly good answer, which is why the filler-phrase rule isn't cosmetic, it's covering a real gap between model latency and human conversational expectations. Setting a backchannel threshold that treats short interjections like 'mhm' as not-an-interruption fixes a bug that shows up constantly in early voice-agent builds without this rule: a user acknowledging what the agent is saying, exactly the way they would in a phone call with a human, accidentally triggers a full barge-in that cuts the agent off mid-sentence over something that was never meant as an interruption. Distinguishing genuine turn-taking from backchanneling by length and word-list match, rather than treating any user audio as a stop signal, is what lets the agent behave the way a competent human phone agent already does without thinking about it.

Verified against

LiveKit Agents 1.0 · 2026-08-05

Deepgram Nova-3 · 2026-08-05

Changelog

  • 2026-08-05 Initial publish, verified against LiveKit Agents 1.0 turn-detection hooks and Deepgram Nova-3 confidence scoring.

Building this for real?

This is a free starting point. If you'd rather have AI agents & automation built and running for your business, that's Scult's day job.

EXPLORE AI AGENTS & AUTOMATION
All AI Agents & RAG prompts

Check your AI visibility

One URL in, a 0–100 score and the exact fixes out.

RUN THE CHECK

Browse all the tools

15 tools across six categories
13 of them never send your data anywhere

Free · No signup · No trial clock

SEE THE DIRECTORY