For anyone building conversational voice What’s been your biggest bottleneck?

AI inference isn’t one product category.

Text LLM serving is a banquet kitchen:
you know the menu early, can batch requests, chunk prefill, and reuse common context through KV caches.

Streaming voice is a roadside dosa stall.

The user changes their order mid-sentence. VAD is separating speech from traffic. STT is revising words live. TTS needs to start speaking in ~50ms or the conversation feels broken.

You cannot “batch” a human interruption.

Yet most voice agents are still built as a thin wrapper around text-generation infrastructure and the lag feels unnatural because the architecture was never designed for real-time turn-taking.

The product question isn’t: “How do we add voice to an LLM?”

It’s: “What does an inference stack look like when latency, interruption, correction, and turn-taking are first-class primitives?”

I think that stack will look very different from vLLM/SGLang-era text serving.

Who is building it?