A multilingual LLM evaluation playground: five interpreters attempt to uncover the meaning hidden inside Indian idioms.

THIS IS Bhaash-phemy! I built Idiom Or Idiot? because I kept noticing a gap: AI models are great at "understanding" Indian languages on paper, but they routinely miss the way Indians actually speak — the idioms, the code-switching, the phrases that only make sense with cultural context. Ask most LLMs to explain a Kannada or Hindi idiom and you'll often get a technically-correct-but-culturally-tone-deaf answer. So I built a community-verified benchmark and game around it. People submit idioms in Kannada and Hindi, models battle it out in blind A/B tests, and an Elo rating tracks which models actually "get it" — not just translate it. Every idiom is manually verified before it counts, so the benchmark stays trustworthy as it grows. My approach evolved a lot along the way — I started with a broader multilingual LLM evaluation framework, then narrowed hard toward something game-like and community-driven, because the fun of guessing/battling turned out to be a much better way to source high-quality data than a dry annotation tool ever could. This is just the start — Kannada and Hindi today, more Indian languages soon. Would love feedback from anyone who's multilingual or just curious how "smart" these models really are about the way people talk. 🙏

@rashmi19 Such an amazing idea! There are so many gaps in AI 'understanding' of languages, one of my linguist-colleagues has written that AI is not multilingual but is multi-monolingual. I would love to follow the further development of your product

@maria_telegina Thank you so much!!!

Huge respect for actually putting this out into the world. Launching something publicly takes courage.

@dora_dao Y'all gave me most of the courage! Thank you so much.

@maria_telegina Thank you so much!!!

Huge respect for actually putting this out into the world. Launching something publicly takes courage.

@dora_dao Y'all gave me most of the courage! Thank you so much.

Idiom or Idiot? — The Oracle of Meaning

OS annotation layer+eval harness for Indian language idioms

OS annotation layer+eval harness for Indian language idioms