Contents
I joined FanTV as an intern and later moved into a full-time software engineering role. One of my tasks was to build a conversational calling agent for sales. The initial problem appeared to be a sequence of familiar components: transcribe speech, generate an answer, and convert that answer back into audio. Testing calls showed that the difficult parts lay at their boundaries. A plausible answer could arrive too late, a brief pause could trigger an unwanted response, and an interruption could leave the agent speaking about a topic the caller had already abandoned.
Latency across the pipeline
The response path includes speech recognition, language-model inference, any required tool calls, and speech synthesis, as well as transport through the telephony provider. In a sequential implementation, delays accumulate along that path. Streaming allows some stages to overlap, so adding their individual durations does not necessarily describe the time a caller waits. I found it more useful to distinguish the time until the first audible response from the time required to finish the whole turn, then inspect where the waiting occurred.
Speech recognition involves decisions before a final transcript is available. Voice activity detection and endpointing determine whether the caller is still speaking, while partial transcripts may change as additional audio arrives. An aggressive endpoint can cut a sentence short; a conservative one adds silence after the caller has finished. This means that two systems using the same recognition model can feel different because their turn-detection settings differ. The model’s transcription speed is only one component of the experience.
Language-model generation and speech synthesis have similar distinctions between starting and finishing. A model that produces its first token quickly may still take time to complete a response, and a speech service may either accept partial text or wait for a complete sentence. Streaming text into speech can reduce the initial silence, but a required database query or external API call can interrupt that flow. Measuring these stages separately helped explain why some turns were consistently quick while others varied with the tools they needed. Audio travelling between the phone, provider, application, and speech services added another source of delay that a model-only benchmark would miss.
Answer quality and context
Once the agent responded quickly enough, testing shifted attention to what it said. Callers asked follow-up questions, changed topics, and referred back to details from earlier in the conversation. A fluent sentence was not enough: the answer needed to use the right information and remain consistent with what had already been discussed. Selecting a capable model helped, but the system also had to decide which parts of the call to retain and what additional information to retrieve.
I tried extracting important entities from the caller’s query and using them to ground subsequent steps. I also experimented with summaries tailored to the domain, preserving details relevant to the sales conversation instead of compressing every turn into a generic account. Another experiment used a secondary model step to anticipate likely follow-up questions and prepare context before the caller asked them. These approaches improved parts of the interaction, but they did not fully solve the problem. An anticipated question can be wrong, and a summary can omit a detail that becomes important later, so neither should replace the actual conversation state.
Deciding when a turn ends
A pause is ambiguous. The caller may have finished, may be searching for a word, or may simply be thinking aloud. Treating every silence as a completed turn made the agent interrupt; waiting through every possible hesitation made it feel unresponsive. Adaptive thresholds were more useful in my testing, allowing additional time when the utterance appeared incomplete and responding sooner when there was a clearer ending. This was a conversational design decision as much as a speech-processing parameter.
One approach was to begin preparing a response at an early pause threshold but withhold playback until a later threshold. If the caller resumed speaking, the prepared response could be discarded. This hides some processing time inside the pause, at the cost of work that may never be used. Turn classifiers offer another approach, although the ones I tried did not consistently handle the Indian conversations in my tests. That experience made it important to evaluate turn detection on representative calls instead of treating a provider’s default as a finished solution.
Interruptions and stale responses
When a caller genuinely interrupts, the application must stop the previous response rather than place the new request behind it. In our implementation, that meant cancelling the language-model stream, stopping speech generation, and preventing late tool results from triggering playback. A new turn needed to supersede the old one throughout the pipeline. Otherwise, cancelling the visible response could still leave background work that eventually resumed speaking with stale information.
Not every sound is an interruption. Expressions such as “haan,” “okay,” or a brief laugh may acknowledge the agent without requesting a new turn. Cancelling on every such sound made the conversation fragment unnecessarily. The system therefore needed to distinguish an acknowledgement from an attempt to take the floor, while also dealing with unrelated speech in the caller’s surroundings. This remained a harder problem than a clean demonstration suggested: a live phone connection contains more than the alternating sentences of a chat transcript.
What the calls taught me
The most useful change in my approach was to evaluate the whole interaction rather than optimize each component in isolation. Faster inference did not repair incorrect endpointing, and better answers did not help if an old response continued after an interruption. Reliable calling agents require latency, context, and cancellation to be designed together. These were lessons from a particular implementation and its tests, rather than a complete solution to conversational AI, but they explained many of the failures that became visible only when people actually used the agent.