A voice bot is software that holds a spoken conversation over a phone call or microphone, with no person on its side of the line. It detects when you stop talking, turns your speech into text, decides what to do, and speaks a reply. An AI voice agent is a voice bot driven by a language model.
Most explanations stop at that pipeline. This guide follows what happens on a real phone line: the time budget for one turn, the mechanics of interruption, what 8 kHz audio changes, why multilingual speech is a harder test, and the rules that apply before an outbound bot dials anyone.
Note: Key takeaways
- "Voice bot" covers a range. Ask who decides the next step: a menu tree, an intent classifier, or a language model. That answer predicts how the bot will fail.
- The clock is the product. One published guide targets about 1,000 ms per turn at the 95th percentile, while human answers commonly start within 200 ms. Transport, transcription, and the model's first token each take a large share.
- Interruption has a memory problem. Stopping the audio is the easy half. The bot must also forget words the caller never heard.
- Test multilingual speech on phone audio. A shared code-switching benchmark on Hindi-English and Bengali-English started at about 32% word errors.
- Rules can cover the automation itself. AI voices count as artificial under US TCPA rules, and India's TRAI now makes senders declare automated calls in advance.
What is a voice bot, and how is it different from a voice agent?
The terms overlap, so define them by behaviour. Voice assistant, voice AI, voice bot, and conversational IVR often get treated as one family. A useful line runs inside that family: voicebot is the broad term that includes scripted systems, while "AI voice agent" usually implies flexible reasoning, multi-turn state, and tools.
The more useful test is who chooses the next step:
| Type | Who decides the next step | What the caller can do | How it usually fails |
|---|---|---|---|
| Touch-tone IVR | A designed menu tree | Press keys | The need isn't on the menu |
| Speech-enabled menu | A menu tree plus a phrase recogniser | Say listed phrases or press keys | It mishears a phrase, or the caller goes off-list |
| Intent-based bot | A classifier plus dialogue rules | Speak freely within trained intents | The request falls outside the trained set |
| LLM voice agent | A language model, bounded by a prompt and tools | Speak freely, across several turns | A confident wrong answer, or the wrong tool call |
| Speech-to-speech agent | One model that takes audio and returns audio | Speak freely, sometimes while it talks | The same, with fewer stages you can inspect |
If you know IVR systems, the top two rows will look familiar. A chatbot can share the reasoning of the lower rows, but it never deals with silence, overlap, or a noisy line. Consumer assistants on a phone or speaker are a different product again, because a business voice bot usually has one narrow job and access to your systems.
How does a voice bot handle a phone call?
Two layers do the work. The telephony layer gets audio to and from the caller. The speech and reasoning layer runs once per turn.
The four stages each have one job and one common way to fail:
- Turn detection. It decides that the caller has finished. LiveKit's documentation describes the options, from plain voice activity detection to a model that weighs both the meaning and the sound of speech. A trigger-happy setting talks over people, and a cautious one leaves dead air.
- Speech to text. It produces a running transcript. A wrong account number here poisons everything downstream, so transcription quality on telephone audio deserves its own test set.
- Agent logic. Rules or a language model choose the reply and any tool call. A slow tool stalls the whole turn, because the caller hears nothing while it runs.
- Text to speech. It generates audio, ideally streaming from the first words of the reply. If it reads $150,000 digit by digit, the caller loses trust quickly.
OpenAI's voice agent guide lists three architectures, and two of them frame the main choice. A chained pipeline suits you when each stage must be visible or replaceable, for example to run a policy check on text before the bot speaks. A single speech model gives you fewer parts to connect and fewer points at which to intervene. The third, a full-duplex live model, keeps talking while a separate backend does the work.
Where does the time go in a single turn?
Time starts when the caller stops speaking and ends when they hear the bot. This is the turn gap. Deepgram's guide suggests a 95th percentile turn under about 1,000 ms and gives planning figures for each stage. Those figures use different measurement boundaries, so treat them as a starting yardstick rather than a sum.
Two findings from conversation research show why this gap matters:
- People answer fast. Across ten languages, response timing to questions peaks between 0 and 200 ms. That sample didn't include an Indian language, so treat it as a broad human baseline.
- Silence carries meaning. Corpus work links gaps of 700 ms or more with disagreeing or declining responses. A bot that always pauses for a second can sound reluctant, even when its answer is yes.
These levers move the budget most, so spend your effort there:
| Lever | What to do | The trade-off |
|---|---|---|
| End-of-turn detection | Replace a fixed silence timer with a model | Cutting the timer alone means interrupting people mid-thought, which feels worse than waiting |
| Model first token | Choose by time to first token, then stream | A faster model may reason less well, so split hard reasoning into a backend call |
| Tool calls | Speak a short bridge such as "let me check that", and set a timeout on every tool | Deepgram's guide cites work putting tool calls at 30 to 80% of latency before the final answer starts |
| Placement | Keep media, speech, and reasoning services close together and near your callers | Regional capacity for a given model may be limited, so check it early |
Measure the slowest turns as well as the median. A caller remembers the four-second silence, not the average. Record each stage separately, or you can't tell a slow model from a slow network.
How does barge-in work?
Barge-in means the caller can talk over the bot and be heard. It's the feature that separates a conversation from a recording, and it's harder than it looks because not every overlap is an interruption.
In a sample of 200 overlaps from a corpus of English telephone conversations, 73% involved a backchannel such as "yeah" or "mm-hm". Those sounds mean "keep going". A bot that stops at every one will stall constantly. LiveKit's adaptive interruption handling targets exactly this distinction, and the same page covers false positives, when noise triggers a stop and no words follow.
The memory step is the one teams miss. The model generates faster than the phone plays, so its history can hold a sentence the caller never heard. On a managed platform, the platform owns this problem. If you own the pipeline, track playback position with markers and rewrite the history at the point of interruption.
What does phone audio change?
A voice bot demo on a laptop microphone flatters every stage. A phone call is harder in four ways.
The audio is narrow. G.711, the long-standing telephone codec, samples at 8 kHz, passes roughly 300 to 3,400 Hz, and runs at 64 kbps. Consonants such as "s" and "f" become hard to separate, and spelled-out names suffer most. Send your speech engine what the call actually gives you, and don't assume a higher upstream sample rate adds detail that the phone leg removed.
The transport has a character. SIP calls carry audio over RTP, usually on UDP, where a lost packet simply disappears. WebSocket streams run over TCP, where a lost packet is resent and later audio waits for it. Neither is free: one loses a fragment, and the other can add jitter on a poor network.
The stream has a bandwidth cost. Base64 text adds a third to the raw size, and slightly more once each frame is padded. See the streaming docs for supported formats. We calculated the load for one direction of one call, at 20 ms frames, before JSON and TLS overhead:
| Format | Raw bytes per 20 ms | Raw rate | With Base64 | For 100 concurrent calls |
|---|---|---|---|---|
| μ-law, 8 kHz | 160 | 64 kbps | 86.4 kbps | 8.6 Mbps |
| L16, 8 kHz | 320 | 128 kbps | 171.2 kbps | 17.1 Mbps |
| L16, 16 kHz | 640 | 256 kbps | 342.4 kbps | 34.2 Mbps |
| L16, 24 kHz (outbound only) | 960 | 384 kbps | 512 kbps | 51.2 Mbps |
Each call holds more than a channel. A bot call occupies a telephone channel and also an open speech-to-text session, a model stream, a text-to-speech stream, and server memory. Size channels as you would for any line: 1,000 calls in a busy hour at 3 minutes each is 50 erlangs, which needs 64 channels at 1% blocking by our Erlang B calculation. The SIP trunking guide walks through that math. Then check that your speech and model providers allow the same concurrency, because their limits often arrive first.
Outbound bots face one more phone-only problem: voicemail. A bot that greets on answer talks over the machine it's trying to classify. Answering machine detection works better when the bot stays silent until the result arrives.
Why is multilingual speech a harder test?
Callers in many countries move between languages inside one sentence, such as Spanish and English in the US or Hindi and English in India, and an off-the-shelf demo rarely shows what that costs.
The evidence is sobering. The MUCS 2021 challenge, built on Hindi-English and Bengali-English code-switched speech among other sets, reported a baseline of 32.45% word error on its code-switching test. A later paper on the same problem reached 21.77 on Hindi-English and 28.27 on Bengali-English. That's still roughly one word in four or five, and these are older research systems, so treat them as a warning and test on your own calls.
Build your evaluation around what breaks in real deployments. These examples come from India, where code-switching is common:
| What callers do | Why it breaks | What to test or change |
|---|---|---|
| Mix Hindi and English: "mera EMI date change karna hai" | One-language models lose words at the switch point | Score on recorded phone calls from your own audience, with both scripts accepted in the reference |
| Say numbers in another language or as "double five" | One wrong digit voids a phone number, amount, or date | Read the value back and confirm, or switch to key presses |
| Use lakh and crore | A reply read in millions confuses, and a wrong amount misleads | Normalise amounts before text to speech, and listen to the result |
| Speak from a street or a shop | Other voices can trigger barge-in | Test noise cancellation and a stricter interruption threshold |
| Have a regional name or locality | Speech engines guess at rare words | Add expected terms as hints where your engine supports them, and confirm spellings |
Key presses still matter here. RFC 4733 carries DTMF as events rather than speech, so a keypad entry skips speech recognition altogether. Use it for a PIN, a date of birth, or any value the caller shouldn't say aloud near other people.
Which rules apply to a voice bot?
Rules increasingly cover the method of calling as well as its content. They change often, so confirm the current text and take legal advice.
AI voices count as artificial in the US. The FCC confirmed in February 2024 that "artificial or prerecorded voice" limits cover current AI-generated voices, which brings those calls under TCPA consent rules.
The trigger is usually who starts the call. An outbound voice bot is an automated call under most of the rules we read. An inbound bot that only answers calls generally isn't, because the definitions turn on who starts the call. That's our reading, so confirm it with your counsel.
Say that it's automated. Confirm with your counsel whether a rule requires business calls to say they're AI. We recommend the practice either way: name the business, say it's an automated assistant, and offer a person early.
Recordings and transcripts are personal data. Both usually identify a person, so plan notice, consent, retention, and deletion under laws such as the EU's GDPR.
If you're in India, TRAI's Third Amendment makes senders declare automated calls and caller ID ranges in advance and treats undeclared ones as unsolicited, promotional calls use the 140xx series (TRAI), and the DPDP Rules phase in to about May 2027. Our SIP trunking guide covers the wider penalties.
Which kind of voice bot should you build?
Start with the simplest thing that does the job. Each step down this map adds cost and adds ways to fail, so a generative model isn't automatically the upgrade.
The last two branches split on ownership, and SIP vs WebSockets compares them in detail:
| Managed platform over SIP | Own pipeline over WebSocket | |
|---|---|---|
| Turn-taking and interruption | The platform handles them | You build them |
| Model choice | What the platform supports | Anything you can connect |
| Scaling | Add channels, and the platform scales its own side | You scale workers as well as channels |
| Existing PBX | SIP fits naturally | No standard bridge |
| Best for | Reaching production quickly | Custom logic, tight data control, high volume |
What goes wrong after launch?
A bot that passes a demo can still fail in three ways. Plan for each one before go-live.
It says something false. A generated answer sounds equally sure whether it's right or wrong. Keep facts such as balances, dates, and policies in tools, and let the model phrase them rather than recall them. Run a check on the text before speech when the wording carries legal weight.
It gets talked into something. A caller's speech is untrusted input to your model, so direct prompt injection, as OWASP defines it, applies to a phone line too. Give each tool only the access it needs. Enforce identity checks and limits in your backend rather than in the prompt, because the model shouldn't be the last line of defence.
It looks successful while callers suffer. Containment measures how many calls never reached a person, and containment targets can discourage human access. Read the numbers together:
| Metric | What it tells you | The trap |
|---|---|---|
| Task completion | Did the booking or update actually happen? | Check the system of record, not the transcript |
| Containment | How many calls stayed with the bot | High containment can mean no way out |
| Repeat contact | Did the caller come back within days? | The honest check on containment |
| Turn gap at the slowest turns | How the worst moments felt | Averages hide them |
| False interruptions | How often the bot stopped for nothing | Only visible if you log stop events |
| Handoff success | Did a person pick up, with context? | A transfer into an empty queue isn't a success |
Published containment and cost-saving ranges rarely state the call mix behind them, so set your own baseline from a small pilot.
Vobiz docs for this topic
The setup details for this are in the Vobiz docs:
- Vapi
- Retell AI
- ElevenLabs
- LiveKit
- SIP trunk
- Streaming call audio
- Your WebSocket server
- Pipecat
- Transfer a Call API
- Machine detection
- Concurrency
- Calling regulations
Anything that places calls or sends messages is billed and reaches real people, so run it only with your own credentials after confirming the details.
Frequently asked questions
What is a voice bot in simple words?
A voice bot is software that talks with people over a call or microphone. It listens, works out what you want, and answers aloud. Simple bots follow a fixed script. Advanced ones use a language model and can act, for example by booking an appointment or looking up an order.
How does a voice bot work?
Each turn runs through four stages. The bot detects that you have stopped talking, converts your speech to text, decides on a reply or action, and converts that reply to audio. A telephony provider carries the audio between the phone network and those stages.
What is the difference between a voice bot and an IVR?
An IVR walks callers through a menu that someone designed in advance, usually with key presses. A voice bot lets callers state their need in their own words and works out the next step. Many voice bots still accept key presses for sensitive input such as a PIN.
Is a voice bot the same as an AI voice agent?
Not quite. Voice bot is the broad term, and it includes scripted systems. AI voice agent usually means a bot built on a language model that keeps context across turns and calls tools to finish tasks. Vendors use the words loosely, so ask who decides each next step.
Are AI voice bots legal?
We found no general ban, but outbound bots carry duties. In the US, AI voices fall under TCPA consent rules. If you're in India, TRAI's Third Amendment treats automated calls as A2P calls that senders must declare in advance, and consent, Do Not Disturb, and number series rules also apply. Check the current text and take legal advice for your case.
How fast should a voice bot respond?
Aim as low as you can without cutting callers off. Human answers commonly start within 200 ms, and one published guide targets a 95th percentile turn under about 1,000 ms. Measure from the caller's last word to the bot's first sound on real calls, including your slowest turns.
Further reading on Vobiz
- AI voice agent solution overview · Build an AI voice agent · All integrations
- Voice streaming over WebSockets · SIP vs WebSockets · SIP vs WebSocket visualizer
- What is an IVR? · What is a voice API? · Intelligent call escalations
Sources
Indian regulators
- TRAI, Press release 119/2026 on the Third Amendment, 18 September 2026.
- TRAI, TCCCPR Third Amendment Regulations, 2026, 18 September 2026.
- TRAI, Press release 91/2026 on the 1600 and 140 series, 10 July 2026.
- RBI, Outsourcing of Financial Services: Responsibilities of regulated entities employing Recovery Agents, 12 August 2022.
- RBI, Amendment Directions on conduct in recovery of loans and engagement of recovery agents, press release, 6 August 2026.
- PIB, Government notifies DPDP Rules to empower citizens and protect privacy, 14 November 2025.
US regulator
- FCC, Declaratory Ruling on AI technologies and the TCPA (FCC 24-17), February 2024.
Standards and security
- IETF, RFC 3550: RTP, July 2003.
- IETF, RFC 4648: Base16, Base32, and Base64 Data Encodings, October 2006.
- IETF, RFC 4733: RTP Payload for DTMF Digits, Telephony Tones, and Telephony Signals, December 2006.
- IETF, RFC 6455: The WebSocket Protocol, December 2011.
- OWASP, LLM01: Prompt Injection.
Reference
- Wikipedia, "G.711".
Research
- Stivers et al., Universals and cultural variation in turn-taking in conversation, PNAS, 2009.
- Levinson and Torreira, Timing in turn-taking and its implications for processing models of language, Frontiers in Psychology, 2015.
- Diwan et al., Multilingual and code-switching ASR challenges for low resource Indian languages, 2021.
- Chadha et al., Code Switched and Code Mixed Speech Recognition for Indic languages, 2022.
Vendor documentation and guides
- Deepgram, Real-time voice AI stack for agents: Architecture guide.
- LiveKit, Turn detection and interruptions.
- OpenAI, Voice agents guide.
Industry summary
- Risk Management Association of India, RBI's new loan recovery directions 2026.
Connect your voice bot to Vobiz — Put your voice bot on a real phone number with a SIP trunk or a WebSocket stream.
If you're putting your voice bot on a real phone number, Vobiz provides AI-native SIP trunks, numbers and a voice API for it.




Top comments (0)