Scientific publishing is broken. Peer review takes months, reviewers are underpaid and overloaded, and the publish-or-perish culture along with AI-assisted writing leads to a flood of daily preprints (almost 1,000 preprints on arXiv.org alone). The result: important papers get buried, mediocre ones get accepted at top venues, and researchers spend more time gaming the system than doing science. Kurate is inspired by LMArena's approach to ranking LLMs through head-to-head comparisons. How it works: Every new arXiv preprint from multiple scientific disciplines gets an AI-generated impact assessment, then papers are paired up and three LLM judges (GPT-5.4, Claude Opus 4.6, Gemini 3 Pro) vote on which has higher scientific impact. Thousands of these pairwise matches produce ranked leaderboards with Elo-style scores. See https://kurate.org/methodology for more details. Claude Opus 4.6 dominates as both summarizer and judge. It's the only model that uses the full 1–10 score range (GPT and Gemini cluster everything at 7–8) and creates very few inconsistencies through intransitive loops (from A>B and B>C it should follow A>C). See https://kurate.org/correlation for more. First validation tests on ICLR datasets have shown promising correlation (ρ = 0.6–0.7) with human reviewers and program committees. See the ongoing validation at https://kurate.org/validation?v=... Kurate and the validation in particular are still a work in progress. Would love feedback, especially from researchers. Does this match your intuition about paper quality?

Kurate.org

AI-powered scientific paper rankings

AI-powered scientific paper rankings