Today, I'm very excited to launch something close to my heart: Introducing, IndiaSocialBench. An eval built from scratch to measure how well a language model can understand Indian context. Whenever a model releases, there is always a talk about coding capabilities & general/emotional intelligence. However, there is no eval to measure how good is that model with Indian context. The IndiaSocialBench is a honest attempt at that. Built around 18 scenarios, across 8 social dimension totalling 50 datasets. Each model go through all the 50 datasets, and thus a detailed analysis will be made leading to final leader board. Currently, started with 3 languages: English, Hindi & Hinglish. Will be expanded to other Indian languages gradually. I love evals, and had a lot of fun building this. Very excited to open the public beta to everyone, and looking forward for feedback & suggestions.

IndiaSocialBench

Does your model understand India?

Does your model understand India?