Viktor.comAn AI coworker that actually does the work
oqoqo
Build evals and custom benchmarks for real-world tasks
722 followers
Build evals and custom benchmarks for real-world tasks
722 followers
Run eval experiments at scale in realistic environments. Define custom task sets to build your private benchmarks, measure how well agents can use any product, and find best models for your use cases. Generate dynamic insights to detect frictions in product interfaces or token inefficiencies.
Free Options
Launch Team / Built With
Have you noticed big differences between Claude Code, Codex, Cursor, and Copilot when running the exact same real world task?
congrats @margharitha & team!
oqoqo
@hamza_afzal_butt definitely. There are a lot of cool insights you can find by doing such running against multiple agents. For example we have found that codex tends to spend a lot more time researching than implementing but ultimately finds the right answer whereas claude code tends to implement and iterate a lot more until it finds the right solution.
Bababot
Can teams compare different models on the exact same custom task set?
oqoqo
@aarav_pittman Yes, ofc! And it's not just for model comparison but you can compare any agent/harness. Even within a given harness, you can compare models and even within a model, you can compare even amongst different reasoning levels/efforts. An experiment is the same custom task set spanned across all these permutation/combinations.
In the FAQ, you might consider changing "The free plan adds runs to your organization each month" to "The free plan resets runs for your organization each month" ("refreshes" could also work).
Adding runs, but runs not rolling over is slightly confusing.
Neat platform though. I can see how this would make it easier to do more sophisticated eval-ing without the hassle of building scaffolds and boilerplate from scratch.
oqoqo
@thedatadavis Aah these are the tiny details that do get missed out in the process of the launch. Thank you so much for pointing out the mismatch. Will be updated and fixed soon!
The part eval systems often miss is recovery behavior: permission denial, stale credentials, partial side effects, and a rerun after failure. A benchmark that scores the happy path but not cleanup and recovery can reward an agent that looks finished while leaving the product in a worse state.
oqoqo
@krekeltronics Definitely! One thing we have found helpful is to add negative rubric criteria that makes sure we are measuring when things go wrong as well.
Sulu
This is super useful! I’ve been building agents and skills to make product onboarding easier for enterprise customers but right now its kind of a black box - I don’t really know how they are using it, where things are breaking and what I should fix first. If I get to see how the agent behaves across diff scenarios and where users are getting stuck it will be huge. Can't wait to use the CLI and run this on autopilot!
oqoqo
@priyansh_rastogi yess! try out the plugin for claude code. It is so much fun to just let claude handle the setup and experiment runs. Also really helpful to create custom visualizations.
Coldtea.ai
The idea of testing agents on the real world tasks make sense. How long does it take to setup an eval with oqoqo ?
oqoqo
@farhan_nazir55 congrats on your launch last day Farhan! you can hook up our plugin/MCP with your coding agent and allow the agent to create eval sets. This makes it really easy to get started. The hardest part of the process is making sure the instructions and rubrics that don't overexplain things to agents.
Serand
oqoqo
@rukhsar_amjad The criteria can be as elaborate/simple as you want, having criteria around quality definitely helps manage this.