I built RepoGym around a question that gets lost in agent benchmarks: can we trust the task itself? A broken baseline, a leaked test or a flaky grading command can make an agent score look meaningful when it is not. RepoGym puts the repository snapshot, test expectations, constraints and reference solution into a task you can inspect and rerun. Start with the included examples, validate the baseline and golden solution, then plug in your own agent. The library records trajectories and produces a standalone HTML report. It is free and MIT licensed; optional model providers have their own costs. I would love feedback from people building coding-agent evaluations: which validation failure has cost you the most time, and what would make this fit your repository? Source: https://github.com/shi1720/repogym Docs: https://shi1720.github.io/repogy... More of my work: https://shivamgupta.web.app/ Connect: https://www.linkedin.com/in/shiv...

RepoGym

Validate the task before you benchmark the coding agent

Validate the task before you benchmark the coding agent