I joined FanTV during summer as an intern. Later, that internship got converted into a full time role.
As an SDE, my first task was to build a conversational calling agent for sales.
This post is a story of what went wrong, what surprised me, and the patterns that helped me make the agent feel reliable. Also, less embarrassing when I used it to call my friends.
Part 1: Latency
The main difference between a voice agent and a text agent is latency. It is also the most obvious challenge. If the agent is supposed to talk like a human, it cannot take forever to respond.
In a voice agent, latency is not one thing. It is a full pipeline:
First, you have STT (speech to text). Audio comes in, you chunk it, you run speech recognition, and you get partial words and partial sentences. If STT is slow or inaccurate, the whole system starts on the wrong foot.
Then comes the LLM step. You take the transcript, add the context, add system instructions, and ask the model what to do next. Some setups do extra steps like intent classification or tool argument extraction, which adds more delay.
If the agent needs information, it calls tools. This is where things get scary, because tools are real APIs. Databases can be slow, external services can time out, and sometimes you need multiple calls. Tool latency quickly becomes the dominant part.
Finally, you do TTS (text to speech). The agent output is converted to audio, and then you stream it back to the user. If your TTS is slow, the user feels like the agent is thinking, even when the answer is ready.
So the real end to end delay is basically:
Total Latency = STT Latency + LLM Latency + Tools Latency + TTS Latency
And you feel the pain even more because conversation has turn taking. If the user interrupts, you also need to cancel the old run, otherwise you end up speaking a stale answer.
Now the tricky part is, latency has two meanings.
One is the actual end to end latency, the time from user speaking to the agent speaking back.
The second is the felt latency, how fast it feels to the user. Streaming can make the agent feel fast even if the total work is not tiny, but only if you design it correctly.
1. STT latency (speech to text)
STT is not just “run speech recognition and get text”. You usually have voice activity detection, endpointing, and partial transcripts. If endpointing is too aggressive, you cut the user mid sentence. If it is too conservative, the agent waits too long and feels slow.
This is why two systems with the same speech model can still feel totally different.
2. LLM latency
It usually has two phases:
- time to first token (how long it takes to start responding)
- time to generate (how long it takes to finish the response)
For real time agents, time to first token matters a lot. If the user hears nothing for 2 seconds, they assume the agent is stuck.
Prompt size matters too. If you keep dumping huge chat history every turn, you pay in both latency and cost.
3. Tools latency
Tools are the biggest real world pain because they are synchronous waiting. The model might be fast, but if it needs to call a database, search, or an internal API, you are blocked until the tool returns.
This is where you see timeouts, retries, and partial failures. Tools are usually the reason the agent feels “randomly slow”.
4. TTS latency (text to speech)
TTS also has a time to first audio (how long it takes to start speaking). Some TTS systems can stream audio as text arrives. Some need the full sentence first. That difference completely changes the experience.
If you can stream TTS, you can start speaking while the model is still generating (that is the best case).
The streaming vs sync reality
In a perfect world, the agent does STT streaming, LLM streaming, and TTS streaming, so the user hears something quickly and the rest fills in naturally.
But the moment you add a blocking tool call, the stream pauses. That is why the best agents either avoid tools for simple turns, call tools early, or say one short line before the tool call and then speak the result.
So instead of just measuring total time to finish the turn, we need to measure time to first word (TTFW) by STT, time to first token (TTFT) by LLM, and time to first audio (TFTA) by TTS.
Total Latency = TTFW + TTFT + TFTA
Calling adds latency too
If we are building a calling agent, there is extra latency that does not show up in a simple STT plus LLM plus TTS diagram.
There is a delay for audio to reach STT, because the audio travels from the user phone to the telephony provider, then to our servers, and then into the STT stream. And there is also a delay from TTS back to the user, because the generated audio has to go out again through the same pipeline.
So choosing the right telephony provider and the right streaming protocol is crucial.
Part 2: Quality of answers
Once you solve the latency and your calling agent starts feeling low latency, you do the obvious thing.
You call your friends to flex in front of them.
That is when the next obvious challenge hits you. Quality of answers.
Because now the agent is fast enough that people actually use it. And when people use it, they test every edge case, they ask follow ups, they change topics, or ask the agent to find a girlfriend for them, and they still expect the agent to stay correct and consistent like a real human.
For me, answer quality was not just about getting the final sentence right. It was about being correct, being grounded in the right data, not hallucinating confidently and still sounding helpful in a real conversation.
At the start, I assumed if we pick a good model, quality will be good. But it is not that simple.
You need a model that is fast enough to respond quickly, can maintain context through a 10 minute call without forgetting what happened in the first minute, and still sounds like a real human.
How to solve this completely?
If you know, please tell me. I will pretend I discovered it first.
Few of the solutions I tried:
- Create an entity extractor to pull important entities from the user query and use them to ground the answer.
- Use domain specific summarization so we keep only the useful parts of context, instead of dumping a generic summary every time.
- Use a secondary LLM step to predict likely next turns and pre load relevant context and tools before the next user message arrives.
This only partially solves the problem. It is still far from perfect.
Part 3: Pausing and interrupting
Once quality becomes decent, the next thing you notice in calls is something very human.
People pause, people think while speaking, people say “umm” and then continue, and people interrupt you the moment you start talking.
If you treat a call like clean turn by turn chat, your agent will feel dumb. Also it will feel awkward, like those robots in movies that answer after a long silence and stare into your soul.
Pauses are not the end of a turn
In real calls, a pause can mean anything.
It can mean the user is thinking, it can mean they are searching for a word, it can mean they got distracted, or it can mean they are done.
So the question becomes: when do you decide the user is finished?
This is where endpointing and silence thresholds become product decisions, not just engineering parameters.
If you cut too early, the agent interrupts the user and feels rude.
If you wait too long, the agent feels slow and awkward.
The best approach I found was to be adaptive. Shorter thresholds for confident endings, longer thresholds when the user sounds like they are still forming the sentence.
But there are other ways to handle pauses:
-
Expensive but fast: set up two thresholds, start and end of pause. The moment you hear the start of a pause, you trigger the pipeline and prepare the answer. You only play the audio if the user stays silent till the end threshold. If they speak again before that, you cancel and start a new run. This way you hide pipeline latency behind the pause.
-
Use a turn classifier to detect end of turn and decide when to respond. Many STT providers ship one, but it generally fails in Indian conversations. You can train your own classifier with your own data.
Interruptions need cancellation, not queuing
If the user interrupts while the agent is speaking, you cannot just queue the new request behind the current run.
You have to stop the old run. Otherwise the agent keeps talking about the old topic, while the user already changed the topic. That is how you lose trust fast.
So we treated every run as cancelable:
- cancel the LLM stream
- stop TTS generation
- ignore any tool results that come back late
- start a new run with the new user query
And most importantly, never let an old run speak after a new run started.
But this leads to a new problem, and it shows up a lot in Indian conversations.
People do not only interrupt to change the topic. Sometimes they make small sounds to show they are listening or thinking.
Stuff like “hmmm”, “haan haan”, “okay”, “acha”, or even a small laugh.
If your system treats these as a brand new user turn, you cancel the agent in the middle for no reason, and the call becomes a comedy.
So you now need to handle two things at the same time:
- first check if the user is just filling the silence, or they are actually interrupting to change the topic
- if they are interrupting, cancel the old run and start a new one
- if they are just using filler words, ignore them and continue the current run
Once you solve all this, your calling Jarvis is basically ready.
It will happily listen and answer a random discussion going on in some uncle’s home because he forgot to cut the call, and your agent is just listening to their family fight and giving opinions while no one is even hearing it.
That is the moment you realise you did not build an AI assistant.
You built a very confident participant in Indian chaos.
Thanks for reading so far. I hope you found it interesting.