... Comment from Enio Aguiar(@wwenioaguiar): @gbasovv  That's a great example. "A signal only means something next to a case where you already know the answer" is the rule I wish I'd started with. The merged chorus is also a nice test case to keep: once you have one known failure, every new checker model has to catch it before it gets the job. Comment from Enio Aguiar(@wwenioaguiar): @farzam_habibi Agreed, that's why I dropped timing ... ... part I would copy. I had a smaller version of it choosing a checker model for my song app: a model listens to the finished song and reports which lyric lines were actually sung. On a song where two chorus

... Subhash, the maker of Chorus. Chorus

... George Basov(@gbasovv): Mine from a song generator: the lyrics a music model returns are what it planned to sing, not what it sang. On three test songs every one drifted, a line dropped, a line sung twice, two chorus lines merged with a word lost, while the screen showed the plan. So now a stronger model listens to the finished track and reports what was actually sung; a faster one was tried for that job and missed the merged ... ... chorus

... expect. In my song app the music model returns the lyrics it planned to sing, and on Russian test songs every one drifted from that plan: a line dropped, a line sung twice, two chorus lines merged. Showing the plan would have been showing what the tool intended, not what it did. So a step now listens to the finished track and the screen shows what was actually sung. Curious about your side: with on device recognition ...

... that is the plan, not what was actually sung. So the check is a second model that listens to the finished track and compares. I tried a smaller, cheaper model as that checker first, and it missed two chorus lines that had been merged into one, so the larger model does the listening now. My takeaway for evals in general: grade the artifact, not the model's description of it, and test the grader on a known bad case before ...

... lyrics are written by a cheap Flash model and that is fine. The step that checks what the singer actually sang against the planned lyrics runs on Pro, because Flash was tried there and missed a merged chorus that Pro and an independent transcription both heard. One known miss on the exact job settled it faster than any benchmark. The other thing: the biggest saving I found was not a tier at all. My music vendor charges $0.30 a song ...

... that cost more in the end? Comment from Kisang Pak(@playpakman): @konstantin_tikhaev  Did you use different models for testing and fix to prevent model bias? If so, did you use different different company (e.g. Claude vs. OpenAI) or different models within the same company (e.g. Opus vs

... model living in the same model picker, which ran all six answer files itself. Humans did exactly one job in this pipeline: invigilation. Final score:  90 to 70. The winner also used 15% fewer output tokens (11,702 vs 13,799).  If you don't trust an AI's grading all six raw answer files are attached below, unedited, and the code runs on click. Grade them yourself. See you in the comments. EXAM NOTICE Grok 4.6 moved ... ... make the beauty visible and explain it like a person? The Registrar (a capybara) Q IDraw a capybara with a hundred circles(Fourier series 30 pts) Fourier told the world that any closed curve can be decomposed into a chorus

... founders, sales leaders, and anyone working in AI or SaaS. Happy to connect with other founders, builders, marketers, and curious people here on PH! Comment from akram abbas(@akram_abbas): Hey Vu! Congrats on launching Insyghtful I automated a competitor report on Gong, Chorus

... work lives (no new password needed ). Fun fact: Review started as a what if we just made this free? conversation in a sprint retro and now here we are gearing up for Product Hunt. I won t join the chorus of new Hunters looking for tips (though you know, sling them my way if you have them). Instead, I ll ask you this: what s the little idea you had that turned out the biggest? And how did the retro ...