Ellis - AI notes for in-person meetings
by•
Ellis is an AI notetaker for in-person meetings. Record your meeting, get a clean transcript with each speaker identified, then ask anything — what was decided, what you missed, how it went. No laptop. No extra hardware. Just your iPhone (or Apple Watch).
Replies
How does the speaker identification actually work in practice, especially when people are talking over each other in a real meeting?
Ellis
@zelihay93u thanks with a question. There are multiple steps under the hood.
Users can record a voice profile during onboarding and from settings (voice enrollment).
When a recording finishes, the app uses a diarization process using AssemblyAI, and returns a transcript with multiple speakers.
The app then uses your voice enrollment to find a best match which gets highlighted in the UI
You can then assign yourself as a specific speaker or add names to others if needed.
The decision to make it work on just an iPhone or Apple Watch is genuinely clever. No extra hardware means it'll actually get used in real meetings instead of sitting in a drawer.
Ellis
@hanifekrkir2f Thanks. The benefit of hardware is to alleviate the friction of starting a recording but that means you need to purchase another hardware devise. Benefit of including it in your iOS infrastructure is that you don't need to
The noisy-room question has a nastier cousin: true overlap, two people talking at the same instant. Diarization picks one speaker per frame, so the quieter voice doesn't get mislabeled, its words just vanish, and nothing in the transcript tells you a sentence went missing. That's harder to catch than plain noise because the output still looks clean. When people talk over each other in a fast brainstorm, does Ellis mark the overlap or just pick a winner?
Ellis
@dipankar_sarkar ooh this is a great question. Short answer: Ellis just picks a winner. Diarization returns one speaker + text per utterance.
I'm really curious though, for you is this a problem worth solving using tech or meeting cadence? in other words, does this frequently come up?
For me it comes up in fast 3+ person brainstorms and standups, almost never in 1:1s or sales calls where people take turns. I wouldn't chase true source separation, that's a research problem you don't want to own. The cheap win is honesty: when AssemblyAI hands back a low-confidence or overlapping stretch, drop a small 'crosstalk here' marker instead of a clean line, so I know to trust my own memory for that bit. A confidently wrong transcript is worse than one that admits a gap.
Ellis
@dipankar_sarkar this is great idea. Just to have better understanding for the situation here, can you give me an example of when and how this matters? It is an attribution problem then when you want to give credit, or is it when your own voice get's overlapped? Genuinely curious. Thanks!
The in-person angle is what sets this apart, but recording therapy or doctor visits is also where the privacy question gets sharp. Does the audio and the diarization run on-device, or does the recording get uploaded to a server to transcribe and match against my saved voice profile? And since everyone else in the room never installed the app, is anything about their voice retained, or is it all local to my phone?
Ellis
@hi_i_am_mimo Short answer: server. Nothing about other speakers' voices is retained.
I see you have an interest in local-first apps and privacy. Curious, would this be a non-starter for you? Thanks!
Honestly, server-side is not a non-starter for me on its own — what decides it is the boundary. Given nothing about other speakers is retained, the two things I would want to verify: is my own audio deleted once the transcript exists, or kept for model training, and is the transcript encrypted at rest so only I can read it? A clear processed-then-discarded policy I can actually check is workable even without full local-first.
Ellis
@hi_i_am_mimo appreciate the followup!
In-person is the right wedge, every AI notetaker assumes a Zoom link exists. How do you handle speaker attribution in a noisy room without everyone wearing a mic? That's the failure mode that killed my voice-memo system for coffee meetings.
Ellis
@chielephant oh nice! you built a voice-memo tool yourself?
I'm experimenting with the following setup:
During onboarding users create a voice profile that gets saved to their account (ie. speaker embeddings via Pyannote)
After each recording it uses AssemblyAI's model for diarization and transcript
UI picker to assign yourself and others to the transcript with best match suggestion
Everyone built for Zoom and forgot rooms exist. How does it handle four people around one table with a single phone mic?
Ellis
@chielephant thanks for the question. A few things:
During onboarding you can create a voice profile that gets saved to your account (ie. speaker embeddings via Pyannote)
After each recording it uses AssemblyAI's model for diarization and transcript
UI picker to assign yourself and others to the transcript with best match suggestion
Honestly it's rare by volume but it clusters right where it hurts, the fast back-and-forth when a decision actually gets made, so it feels worse than the raw percentage suggests. I wouldn't chase overlapped-speech separation, it's expensive and still flaky. I'd just surface it: flag the low-confidence diarization spans so I know which two or three lines to re-listen to, instead of trusting a transcript that looks clean. The silent winner-pick is the part that bit me, not the noise.
Ellis
@dipankar_sarkar thanks!
How do you balance giving users AI coaching feedback without making conversations feel overly analyzed?
Ellis
@robert_dimla Can you tell me more what you mean with overly analyzed? Appreciate your time.
From my perspective, I don't think the technology is the issue. It's more about privacy and consent since this is designed for IRL conversations, how do you encourage users to make sure everyone is feeling comfortable being recorded ?
Ellis
@reda_roqai_chaoui at the moment using messaging. There are some other ideas to make it even more explicit, such as a consent toggle. Thoughts?