Hamming AI (YC S24) - Automated testing for voice agents

👋 Hi ya'll - Sumanyu and Marius here from Hamming AI. Hamming lets you automatically test your LLM voice agent. In our interactive demo, you play the role of the voice agent, and our agent will play the role of a difficult end user. We'll then score your performance on the call. 🕵️ Try it here: https://app.hamming.ai/voice-demo (no signup needed). In practice, our agents call your agent! Marius and I previously ran growth and data teams at companies like Citizen, Tesla, and Anduril. We're excited to launch our automated voice testing feature to help you test your voice agents 100x faster than manual phone calls. 📞 LLM voice agents currently require a LOT of iteration and tuning. For example, one of our customers is building an LLM drive-through voice agent for fast food chains. Their KPI is order accuracy. It's crucial for their system to gracefully handle dietary restrictions like allergies and customers who get distracted or otherwise change their minds mid-order. Mistakes in this context could lead to unhappy customers, potential health risks, and financial losses. 🪄 Our solution involves four steps: (1) Create diverse but realistic user personas and scenarios covering the expected conversation space. We create these ourselves for each of our customers. (2) Have our agents call your agent when we test your agent's ability to handle things like background noise, long silences, or interruptions. Or have us test just the LLM / logic layer (function calls, etc.) via an API hook. (3) We score the outputs for each conversation using deterministic checks and LLM judges tailored to the specific problem domain (e.g., order accuracy, tone, friendliness). (4) Re-use the checks and judges above to score production traffic and use it to track quality metrics in production. (i.e., online evals) We created a Loom recording showing our customers' logged-in experience: Logged-in Video Walkthrough We think there will be more and more voice companies, and making the experimentation process easier is a problem we are excited about solving. 📩 If you're building voice agents and you're struggling to make them reliable, reach out at sumanyu@hamming.ai! ❤️ Shoutout to @rajiv_ayyangar and @gabe for helping us with the launch!

Evals for a specific industry is a great idea. Llm as a judge is great, but comes with its own challenges, would be interesting to see how it performs for wide usecases. Also, persona generation automation based on usecases would also be great. Im sure thats in our roadmap 😀 Congratulations on the launch! 🚀

1000 parallel simulated calls to the AI voice agent is such a banger line and claim! As a product person, this is my #1 concern when I build AI product which is consistency. Sometimes, you just don't know why something breaks. Testing by hands and evaluating with eyes only go so far. It's about time for automated testing for LLM voice agent! Does this work with traditional chatbot usage? For example, we have one AI avatar talking to a human through web app? Congrats on the launch @sumanyu_sharma and team!

Couldn't be more excited for these guys!! They have been working incredibly hard and have provided significant value to all of their current customers. They love them and anyone building in voice can get incredibly better results through Hamming!

Hi @sumanyu_sharma congrats on the launch 💥

@sumanyu_sharma Cheers for the launch!! Are the personas fixed or custom to my use case?

Impressive efficiency boost!

Congrats on the launch, this looks like a interesting product.

Hey Sumanyu, I’m curious, how adaptable is Hamming to different industries? For instance, could it be used to test voice agents in customer support or healthcare settings? Congrats on the launch!

"Best demo at YC Live" -- Garry Tan's words, not mine :)

This sounds like a game changer for voice agents! I’m curious about the scalability of the testing. Can Hamming handle multiple LLMs calling each other simultaneously for larger companies? Would love to see a case study on that!

Interesting. Would you be able to also rate the input-output dataset in Post-production? @sumanyu_sharma

Congrats on the launch, 100x faster than manual calls is impressive and will definitely help developers improve their LLM agents. The approach of creating realistic user personas and scenarios is spot on, especially in high-stakes settings like drive-throughs. I can see how this could drastically reduce errors. Excited to see the impact this will have on the voice tech landscape! Upvoted and can't wait to try the demo! Keep pushing the boundaries!

This could save me so much time on testing. How do you handle diverse user scenarios?

This is an impressive way to streamline voice agent testing, Sumanyu! Automating the process and scoring performance is a game changer for anyone in voice tech. The focus

This is a game changer, @sumanyu_sharma! Testing LLM voice agents at warp speed 🌟 is something we desperately needed. I love that you’re not just throwing more data at the problem but actually scoring real-world scenarios. The drive-through example really hits home; it’s all about that order accuracy! Can’t wait to see how this evolves and transforms the industry. Upvoted!

Hamming team, congrats on the launch!

Impressive. This is sure to drive technological advancements in the industry and deliver better voice experiences for users. Congrats on the launch!

This is an impressive launch, @sumanyu_sharma and @marius! The automation of voice agent testing can truly transform how companies evaluate their systems. The detailed analytics you provide will definitely help teams identify issues, especially in complex scenarios like fast-food drive-throughs. I love how you focus on real user personas, as it adds so much value to the testing process. Can't wait to try out the demo and see how Hamming can elevate the quality of LLM agents in real-time! Keep up the great work, Makers! 🚀

Hamming is an impressive tool! It tests your AI voice agents 100 times faster than manual calls, allows you to create Character.ai-style personas and scenarios, runs hundreds of simultaneous phone calls to identify bugs, and provides detailed analytics for improvement. This will greatly enhance testing efficiency and product quality for developers. 📞🤖🚀