Which trace format are you actually on?

by

whatbroke goes live here in about an hour, and there's one thing I genuinely can't answer from my own desk.

It diffs two agent runs and shows what changed between them: which tool calls fired, how the arguments differed, what each run cost, where the outputs diverged. The importer currently reads plain JSONL, OpenTelemetry, Langfuse and LangSmith exports.

Four formats is where I ran out of things I use myself, not where the problem ends. So: what are you exporting traces as right now? If it's none of those four, I'd rather know today than guess at the next importer.

Second question, and this one I'm too close to judge. The output is a terminal report with BROKE and CHANGED blocks, before/after lines for arguments, and a cost and latency delta per case. Is that readable to someone who didn't write it, or does it need a legend?

For context on why it exists: I changed one line in a system prompt and the agent quietly stopped making a refund call. The final message still said the refund had gone through, so the text looked fine, and the eval score moved from 0.82 to 0.79, which told me nothing useful. I found it by opening two trace files side by side and scrolling until my eyes crossed.

npm i -g whatbroke-cli if you want to try it against your own traces before the launch goes up.

2 views

Add a comment

Replies

Be the first to comment