cto bench - The ground truth code agent benchmark
by•
Most AI benchmarks are built backwards. Someone sits down, dreams up hard problems, and then measures how well agents solve them. The results are interesting, sure. But they don't always tell you what matters: how agents perform on the actual work that's sitting in your queue.
That's why we built cto.bench.
Instead of hypothetical tasks, we're building our benchmark from real work. Every data point on cto bench comes directly from how cto.new users are actually using our platform.
Replies
cto.new
TrackerJam
Finally, a benchmark that measures usefulness instead of academic cleverness. This feels much closer to how teams actually decide whether an agent is worth adopting.
cto.new
@maklyen_may thanks! Interesting that OSS models are so high up the list for practical use, eh?
DeepTagger
Wow, this is amazing! All the best models for free! 🚀
How can this be sustainable for you?
cto.new
@avloss great question! We're still working on that. What would you recommend?
DeepTagger
@michael_ludden
Some ideas:
Provide additional services for fee, like Domain, Hosting, Monitoring, Promotion / Ads, Databases.
Charge for organisational use any/or for dedicated deployment.
Charge for additional features, like a human reviewing and solving a problem in case LLM is stuck.
Use collected data to train proprietary models, then sell those.
cto.new
@avloss love it! 🙏
This is a really refreshing take on benchmarks 👀
Grounding it in real work instead of synthetic tasks feels way more honest — as a builder, that’s the kind of signal I actually trust. Love the “built from usage” philosophy. Congrats on the launch! 🚀
Curious how you’re thinking about bias over time — do you plan to balance workloads or surface context around where the data comes from?
cto.new
@elevenapril can you expand on the question a bit more? Not sure what you're asking.
Awesome! Very useful!