DEV Community

Cover image for What Is a Voice Bot? How AI Voice Agents Handle Calls
Piyush Sahoo
Piyush Sahoo

Posted on

What Is a Voice Bot? How AI Voice Agents Handle Calls

A voice bot is software that holds a spoken conversation over a phone call or microphone, with no person on its side of the line. It detects when you stop talking, turns your speech into text, decides what to do, and speaks a reply. An AI voice agent is a voice bot driven by a language model.

Most explanations stop at that pipeline. This guide follows what happens on a real phone line: the time budget for one turn, the mechanics of interruption, what 8 kHz audio changes, why multilingual speech is a harder test, and the rules that apply before an outbound bot dials anyone.

Note: Key takeaways

  • "Voice bot" covers a range. Ask who decides the next step: a menu tree, an intent classifier, or a language model. That answer predicts how the bot will fail.
  • The clock is the product. One published guide targets about 1,000 ms per turn at the 95th percentile, while human answers commonly start within 200 ms. Transport, transcription, and the model's first token each take a large share.
  • Interruption has a memory problem. Stopping the audio is the easy half. The bot must also forget words the caller never heard.
  • Test multilingual speech on phone audio. A shared code-switching benchmark on Hindi-English and Bengali-English started at about 32% word errors.
  • Rules can cover the automation itself. AI voices count as artificial under US TCPA rules, and India's TRAI now makes senders declare automated calls in advance.

What is a voice bot, and how is it different from a voice agent?

The terms overlap, so define them by behaviour. Voice assistant, voice AI, voice bot, and conversational IVR often get treated as one family. A useful line runs inside that family: voicebot is the broad term that includes scripted systems, while "AI voice agent" usually implies flexible reasoning, multi-turn state, and tools.

The more useful test is who chooses the next step:

Type Who decides the next step What the caller can do How it usually fails
Touch-tone IVR A designed menu tree Press keys The need isn't on the menu
Speech-enabled menu A menu tree plus a phrase recogniser Say listed phrases or press keys It mishears a phrase, or the caller goes off-list
Intent-based bot A classifier plus dialogue rules Speak freely within trained intents The request falls outside the trained set
LLM voice agent A language model, bounded by a prompt and tools Speak freely, across several turns A confident wrong answer, or the wrong tool call
Speech-to-speech agent One model that takes audio and returns audio Speak freely, sometimes while it talks The same, with fewer stages you can inspect

If you know IVR systems, the top two rows will look familiar. A chatbot can share the reasoning of the lower rows, but it never deals with silence, overlap, or a noisy line. Consumer assistants on a phone or speaker are a different product again, because a business voice bot usually has one narrow job and access to your systems.

How does a voice bot handle a phone call?

Two layers do the work. The telephony layer gets audio to and from the caller. The speech and reasoning layer runs once per turn.

Architecture of a voice bot on a phone call. A caller reaches a telephony provider through the phone network. A SIP trunk or WebSocket stream carries audio to turn detection, speech to text, agent logic with tools and data, and text to speech, and the reply returns over the same transport. A speech-to-speech model can replace three stages, and a human handoff is the exit.

The four stages each have one job and one common way to fail:

  1. Turn detection. It decides that the caller has finished. LiveKit's documentation describes the options, from plain voice activity detection to a model that weighs both the meaning and the sound of speech. A trigger-happy setting talks over people, and a cautious one leaves dead air.
  2. Speech to text. It produces a running transcript. A wrong account number here poisons everything downstream, so transcription quality on telephone audio deserves its own test set.
  3. Agent logic. Rules or a language model choose the reply and any tool call. A slow tool stalls the whole turn, because the caller hears nothing while it runs.
  4. Text to speech. It generates audio, ideally streaming from the first words of the reply. If it reads $150,000 digit by digit, the caller loses trust quickly.

OpenAI's voice agent guide lists three architectures, and two of them frame the main choice. A chained pipeline suits you when each stage must be visible or replaceable, for example to run a policy check on text before the bot speaks. A single speech model gives you fewer parts to connect and fewer points at which to intervene. The third, a full-duplex live model, keeps talking while a separate backend does the work.

Where does the time go in a single turn?

Time starts when the caller stops speaking and ends when they hear the bot. This is the turn gap. Deepgram's guide suggests a 95th percentile turn under about 1,000 ms and gives planning figures for each stage. Those figures use different measurement boundaries, so treat them as a starting yardstick rather than a sum.

Range chart of one voice bot turn. SIP or PSTN transport adds 200 to 400 milliseconds, speech to text takes 200 to 500, LLM first token is planned at 200 to 400, and text to speech first byte can be as low as 80. Markers show the human response peak at 0 to 200 milliseconds, 700 milliseconds, where long human gaps start to signal reluctance, and a 95th percentile turn target of about 1,000 milliseconds.

Two findings from conversation research show why this gap matters:

These levers move the budget most, so spend your effort there:

Lever What to do The trade-off
End-of-turn detection Replace a fixed silence timer with a model Cutting the timer alone means interrupting people mid-thought, which feels worse than waiting
Model first token Choose by time to first token, then stream A faster model may reason less well, so split hard reasoning into a backend call
Tool calls Speak a short bridge such as "let me check that", and set a timeout on every tool Deepgram's guide cites work putting tool calls at 30 to 80% of latency before the final answer starts
Placement Keep media, speech, and reasoning services close together and near your callers Regional capacity for a given model may be limited, so check it early

Measure the slowest turns as well as the median. A caller remembers the four-second silence, not the average. Record each stage separately, or you can't tell a slow model from a slow network.

How does barge-in work?

Barge-in means the caller can talk over the bot and be heard. It's the feature that separates a conversation from a recording, and it's harder than it looks because not every overlap is an interruption.

In a sample of 200 overlaps from a corpus of English telephone conversations, 73% involved a backchannel such as "yeah" or "mm-hm". Those sounds mean "keep going". A bot that stops at every one will stall constantly. LiveKit's adaptive interruption handling targets exactly this distinction, and the same page covers false positives, when noise triggers a stop and no words follow.

Two timelines. In the first, the caller says mm-hm during playback, and the app keeps playing. In the second, the caller says no, the other one. The app confirms speech, clears queued playback, cuts unheard words from conversation history, and answers the new request.

The memory step is the one teams miss. The model generates faster than the phone plays, so its history can hold a sentence the caller never heard. On a managed platform, the platform owns this problem. If you own the pipeline, track playback position with markers and rewrite the history at the point of interruption.

What does phone audio change?

A voice bot demo on a laptop microphone flatters every stage. A phone call is harder in four ways.

The audio is narrow. G.711, the long-standing telephone codec, samples at 8 kHz, passes roughly 300 to 3,400 Hz, and runs at 64 kbps. Consonants such as "s" and "f" become hard to separate, and spelled-out names suffer most. Send your speech engine what the call actually gives you, and don't assume a higher upstream sample rate adds detail that the phone leg removed.

The transport has a character. SIP calls carry audio over RTP, usually on UDP, where a lost packet simply disappears. WebSocket streams run over TCP, where a lost packet is resent and later audio waits for it. Neither is free: one loses a fragment, and the other can add jitter on a poor network.

The stream has a bandwidth cost. Base64 text adds a third to the raw size, and slightly more once each frame is padded. See the streaming docs for supported formats. We calculated the load for one direction of one call, at 20 ms frames, before JSON and TLS overhead:

Format Raw bytes per 20 ms Raw rate With Base64 For 100 concurrent calls
μ-law, 8 kHz 160 64 kbps 86.4 kbps 8.6 Mbps
L16, 8 kHz 320 128 kbps 171.2 kbps 17.1 Mbps
L16, 16 kHz 640 256 kbps 342.4 kbps 34.2 Mbps
L16, 24 kHz (outbound only) 960 384 kbps 512 kbps 51.2 Mbps

Each call holds more than a channel. A bot call occupies a telephone channel and also an open speech-to-text session, a model stream, a text-to-speech stream, and server memory. Size channels as you would for any line: 1,000 calls in a busy hour at 3 minutes each is 50 erlangs, which needs 64 channels at 1% blocking by our Erlang B calculation. The SIP trunking guide walks through that math. Then check that your speech and model providers allow the same concurrency, because their limits often arrive first.

Outbound bots face one more phone-only problem: voicemail. A bot that greets on answer talks over the machine it's trying to classify. Answering machine detection works better when the bot stays silent until the result arrives.

Why is multilingual speech a harder test?

Callers in many countries move between languages inside one sentence, such as Spanish and English in the US or Hindi and English in India, and an off-the-shelf demo rarely shows what that costs.

The evidence is sobering. The MUCS 2021 challenge, built on Hindi-English and Bengali-English code-switched speech among other sets, reported a baseline of 32.45% word error on its code-switching test. A later paper on the same problem reached 21.77 on Hindi-English and 28.27 on Bengali-English. That's still roughly one word in four or five, and these are older research systems, so treat them as a warning and test on your own calls.

Build your evaluation around what breaks in real deployments. These examples come from India, where code-switching is common:

What callers do Why it breaks What to test or change
Mix Hindi and English: "mera EMI date change karna hai" One-language models lose words at the switch point Score on recorded phone calls from your own audience, with both scripts accepted in the reference
Say numbers in another language or as "double five" One wrong digit voids a phone number, amount, or date Read the value back and confirm, or switch to key presses
Use lakh and crore A reply read in millions confuses, and a wrong amount misleads Normalise amounts before text to speech, and listen to the result
Speak from a street or a shop Other voices can trigger barge-in Test noise cancellation and a stricter interruption threshold
Have a regional name or locality Speech engines guess at rare words Add expected terms as hints where your engine supports them, and confirm spellings

Key presses still matter here. RFC 4733 carries DTMF as events rather than speech, so a keypad entry skips speech recognition altogether. Use it for a PIN, a date of birth, or any value the caller shouldn't say aloud near other people.

Which rules apply to a voice bot?

Rules increasingly cover the method of calling as well as its content. They change often, so confirm the current text and take legal advice.

AI voices count as artificial in the US. The FCC confirmed in February 2024 that "artificial or prerecorded voice" limits cover current AI-generated voices, which brings those calls under TCPA consent rules.

The trigger is usually who starts the call. An outbound voice bot is an automated call under most of the rules we read. An inbound bot that only answers calls generally isn't, because the definitions turn on who starts the call. That's our reading, so confirm it with your counsel.

Say that it's automated. Confirm with your counsel whether a rule requires business calls to say they're AI. We recommend the practice either way: name the business, say it's an automated assistant, and offer a person early.

Recordings and transcripts are personal data. Both usually identify a person, so plan notice, consent, retention, and deletion under laws such as the EU's GDPR.

If you're in India, TRAI's Third Amendment makes senders declare automated calls and caller ID ranges in advance and treats undeclared ones as unsolicited, promotional calls use the 140xx series (TRAI), and the DPDP Rules phase in to about May 2027. Our SIP trunking guide covers the wider penalties.

Which kind of voice bot should you build?

Start with the simplest thing that does the job. Each step down this map adds cost and adds ways to fail, so a generative model isn't automatically the upgrade.

Decision map. If callers' needs fit a few fixed choices, use a speech-enabled menu. If every sentence must be approved in advance, use a scripted intent bot. If you don't need to own each pipeline stage, use a managed platform over a SIP trunk. Otherwise, build your own pipeline over a WebSocket stream. Every branch keeps a human exit.

The last two branches split on ownership, and SIP vs WebSockets compares them in detail:

Managed platform over SIP Own pipeline over WebSocket
Turn-taking and interruption The platform handles them You build them
Model choice What the platform supports Anything you can connect
Scaling Add channels, and the platform scales its own side You scale workers as well as channels
Existing PBX SIP fits naturally No standard bridge
Best for Reaching production quickly Custom logic, tight data control, high volume

What goes wrong after launch?

A bot that passes a demo can still fail in three ways. Plan for each one before go-live.

It says something false. A generated answer sounds equally sure whether it's right or wrong. Keep facts such as balances, dates, and policies in tools, and let the model phrase them rather than recall them. Run a check on the text before speech when the wording carries legal weight.

It gets talked into something. A caller's speech is untrusted input to your model, so direct prompt injection, as OWASP defines it, applies to a phone line too. Give each tool only the access it needs. Enforce identity checks and limits in your backend rather than in the prompt, because the model shouldn't be the last line of defence.

It looks successful while callers suffer. Containment measures how many calls never reached a person, and containment targets can discourage human access. Read the numbers together:

Metric What it tells you The trap
Task completion Did the booking or update actually happen? Check the system of record, not the transcript
Containment How many calls stayed with the bot High containment can mean no way out
Repeat contact Did the caller come back within days? The honest check on containment
Turn gap at the slowest turns How the worst moments felt Averages hide them
False interruptions How often the bot stopped for nothing Only visible if you log stop events
Handoff success Did a person pick up, with context? A transfer into an empty queue isn't a success

Published containment and cost-saving ranges rarely state the call mix behind them, so set your own baseline from a small pilot.

Vobiz docs for this topic

The setup details for this are in the Vobiz docs:

Anything that places calls or sends messages is billed and reaches real people, so run it only with your own credentials after confirming the details.

Frequently asked questions

What is a voice bot in simple words?

A voice bot is software that talks with people over a call or microphone. It listens, works out what you want, and answers aloud. Simple bots follow a fixed script. Advanced ones use a language model and can act, for example by booking an appointment or looking up an order.

How does a voice bot work?

Each turn runs through four stages. The bot detects that you have stopped talking, converts your speech to text, decides on a reply or action, and converts that reply to audio. A telephony provider carries the audio between the phone network and those stages.

What is the difference between a voice bot and an IVR?

An IVR walks callers through a menu that someone designed in advance, usually with key presses. A voice bot lets callers state their need in their own words and works out the next step. Many voice bots still accept key presses for sensitive input such as a PIN.

Is a voice bot the same as an AI voice agent?

Not quite. Voice bot is the broad term, and it includes scripted systems. AI voice agent usually means a bot built on a language model that keeps context across turns and calls tools to finish tasks. Vendors use the words loosely, so ask who decides each next step.

Are AI voice bots legal?

We found no general ban, but outbound bots carry duties. In the US, AI voices fall under TCPA consent rules. If you're in India, TRAI's Third Amendment treats automated calls as A2P calls that senders must declare in advance, and consent, Do Not Disturb, and number series rules also apply. Check the current text and take legal advice for your case.

How fast should a voice bot respond?

Aim as low as you can without cutting callers off. Human answers commonly start within 200 ms, and one published guide targets a 95th percentile turn under about 1,000 ms. Measure from the caller's last word to the bot's first sound on real calls, including your slowest turns.

Further reading on Vobiz

Sources

Indian regulators

US regulator

Standards and security

Reference

Research

Vendor documentation and guides

Industry summary

Connect your voice bot to Vobiz — Put your voice bot on a real phone number with a SIP trunk or a WebSocket stream.


If you're putting your voice bot on a real phone number, Vobiz provides AI-native SIP trunks, numbers and a voice API for it.

Top comments (0)