Launching today
HarnessForge Studio
Replay an AI agent trace offline and find where it broke
2 followers
Replay an AI agent trace offline and find where it broke
2 followers
Import a recorded agent trace (JSON/JSONL), replay it as a timeline, see loops, retries, failed tools and token growth flagged, compare the fixed run against the broken one, and export a report with secrets masked. One offline HTML file, $49 once.
Hi all. I built this because I kept losing evenings scrolling through agent traces trying to work out where a run went sideways. Forty tool calls, half of them retries that look identical, and the token count quietly doubling somewhere in the middle. Reading raw JSONL for that is miserable.
So HarnessForge loads the trace and shows it as a timeline. It pairs each tool call with its result, flags the repeats, loops, failed calls and token spikes, and tells you why it flagged them. You can load the fixed run next to the broken one and see what changed. Then export the whole thing as Markdown, JSON, CSV or text with any API keys it spotted masked out.
It's one HTML file. Runs in Chrome or Edge on your desktop, nothing gets uploaded, no account.
Things it doesn't do, so nobody's surprised: it doesn't run or predict a model, it just reads what was recorded. The warnings are heuristics, not answers. And the secret masking is pattern-based, so check an export before you share it.
can i load two traces from different agents into the compare view or do they need to be the same format?
@qhelani_chix Yes, they can be different formats. Each run is imported separately (OpenAI-style, Anthropic-style, LangSmith, OpenTelemetry, or the field-mapping dialog for anything else), and both get normalised into the same event model before Compare looks at them. So Run A can be one format and Run B another.
One thing to know: Compare aligns steps by shared step ids if both runs have them, otherwise by order + event type + tool name. It's built for "same task, broken run vs fixed run", where that lines up well. Two traces from different agents doing different things will still give you the metrics side by side (tool calls, failures, retries, loops, tokens, duration), but the step-by-step alignment and diff will be rougher because there's no real correspondence to find.