Why does my AI need 2 seconds to answer when the speech-to-text is already done?🤔
I'm building a real-time voice AI application, and I ran into a problem I didn't expect.
The speech-to-text part is already streaming and feels fast⚡. But then...
The user stops speaking → transcript is ready → ...nothing happens for a moment → AI finally starts answering.
That gap is surprisingly noticeable.
My current pipeline is roughly:
Microphone → Deepgram Streaming STT → detect question → LLM → streaming response
And now I'm trying to understand where those extra milliseconds are actually going⏱️. There are potentially several places:
* Deepgram endpointing
* Waiting for the final transcript
* Question/utterance detection
* Network round trip
* LLM time-to-first-token
* Prompt size
* Model selection
* Streaming configuration
* How the different stages are orchestrated
The goal is to make the experience feel almost instantaneous; ideally <1 second from the end of speech to the first useful AI output.
So I'm curious:
1) If you've built a real-time voice AI system, what was your biggest latency bottleneck?
2) And what actually made the biggest difference?
I'm especially interested in real-world numbers:
What end-to-end latency have you managed to achieve?👇
Replies
I feel like that little wait after someone stops talking is probably the biggest part of what makes it feel slow.
I wonder if where everything is running makes a difference too. A few extra trips back and forth could explain some of that delay.
For me, getting the first few words out quickly would matter more than how long the rest of the answer takes.
Honestly, even a quick “got it” while the answer is being worked out might make that 2 second silence feel way less awkward.
interesting!, but with which LLM are you working on? 🤔 That's, what I think, the only thing that makes the latency after text-to-speech has finished