Sun - Collaborative voice API for agents
by•
Sun is a voice first AI model built specifically for real‑time collaborative voice interaction, not just one‑on‑one chat.
ChatGPT Realtime and Gemini Live were built for one user talking to AI. Sun is built for collaboration — meetings, group calls, multi-agent debates, classrooms.
One API, multi-speaker awareness, 10× the context window.
Replies
Does it mean that by using this, we can make our internal meeting agent talk in meetings about what is happening in meetings ? Like talking fireflies or otter ?
@regaldreamtech Exactly — that's the core use case Sun was built for.
Fireflies and Otter listen silently. With Sun, you can build an agent that listens, knows who's saying what, and actually speaks up at the right moment — to summarize, answer a question, surface a related decision from earlier in the meeting, or even moderate.
how does it know when to interrupt and when to not? Also, is there way to make it speak only when explicitly asked?
@ashishkingdom Yes, you can actually give trigger words or names to the agent. The agent will then respond only when called with the name (and is asked a question), not when passively mentioned in a conversation. It also looks at voice activity to see if someone is actively speaking to avoid interrupting them.
Does it take audio in or should we give transcribed text as inputs ?
@jaison1993 Sun takes in text input (both partial and final transcripts - streaming). It gives out both text and audio output.
Sun
@aswanth_viswanathan8 Yes, it is possible to add context mid-session using context.update. It could be useful if you want the agent to have certain information, like sales number or a context about the ongoing discussion, in the middle of discussion.
This is interesting, how many speakers/voices does it tolerate?
@real_digidavid The model doesn't detect speakers, but it takes in speaker name and what they said as a streaming text input. It can therefore support infinite speakers theoretically. We have tested up to 25 participant meetings. The output from the model is both text and audio. The output audio has 4 different voices which you can switch while the session is in progress as well. 2 male and 2 female voices.
So, is this like speech engine for note takers like otter or fireflies ? like making them talk back. fireflies does that now, but it takes forever to make it work.
@arjun_reghu Yes, absolutely. This will help with in-meeting intelligence (like meeting assistant) and is faster than the fireflies version. It can respond in voice in a very short time. For comparison, for a smaller answer, fireflies will start speaking after the sun agent has completed the answer, or is halfway. Take a look into our playground where you can try it out.
@aisha_m_a Yes, the trigger words list is for alternate names of the agent. Agent will see it as a wake up call to answer. It is also helpful if the transcription service used sometimes misses to transcribe properly. For instance, John can be sometimes written as Jon and this could help in those scenarios as well.
The classroom use case is the one that caught my attention. An AI that can follow multiple speakers and participate at the right moment could be genuinely useful during group discussions.
Have you tested it with actual teachers or students yet? I'd be curious to hear how it performs in a real classroom setting.
Congrats on the launch!
@jared_salois Yes, this is build for multiple speakers being present, such as inside a group call or meeting. It is designed to be polite and avoid cutting off people and to allow interruptions. We have tested the interruption and multi speaker handling as well. But if people in the call keeps interrupting the voice, it will wait for them to complete speaking by being polite.
Refocus
The "when to interrupt" part is exactly where most voice infra falls down. I'm building voice AI for older adults, and barge-in plus handling slow or overlapping speech is the hardest piece, models either talk over people or freeze. Curious how Sun handles turn-taking when speakers have very different pacing, and whether latency holds up with 4-5 voices in the room?
@igorgurovich First and foremost, sun has trigger words and name option, which only wakes the model up to answer when required. It monitors transcription streaming to see if someone is speaking, and waits until it is complete (no more streamed text). Then, it decides to speak. And even after this, there could be incidents of people speaking over, which is handled by interruption handling, where model either decides to let the person continue or resumes the talk if the other person interrupted accidentally. Only a combination of these can actually help improve the experience.