When your agents get complex
Simulation-based AI agent testing and evaluation that turns unpredictable agents into reliable production systems.
This is the 4th launch from LangWatch. View more
Claude Code usage tracking by LangWatch
Launched this week
Track Claude Code usage: cost, cache, session replay. Run `npx langwatch claude` once. Every session gets cost with cache reads/writes as separate token classes, every bash and MCP call as a span, theoretical vs billed for your Max plan, and a full terminal replay in the UI. Works for Codex too.
Free
Launch Team
the MCP tool calls as spans is the bit i want most. i drive a browser automation MCP from claude code all day and the screenshots are obviously the expensive part, except obviously is doing a lot of work in that sentence because i have no per-call number for any of it.
cache reads and writes counted separately is the other one, that is where the surprise usually hides.
LangWatch
@alex_watson2110 yeeppp both are in there: every MCP call gets its own span with its own token cost, and cache reads and writes are broken out separately, so you'll have a real per-screenshot number instead of a hunch.
Manouk's "we run a fleet of coding agents ourselves and had no idea what they actually cost" is the honest version of this. Same here — our answer was "we'll find out when the subsidies stop."
One push: cost per session can mislead. The number that changes how you route work is cost per outcome — per merged PR, per green CI run, per fix that actually held. A run that spends 3x on the expensive model and lands it first try beats a cheap model looping four times — but a per-session view shows the opposite, and quietly trains you to downgrade the model that was earning its price.
You're already logging tool calls as spansm that: span → the commit or test run itproduced. On the roadmap, or are you deliberately staying at the session boundary?
+1 to the immutable price-stamp point above, with a wrinkle: it has to cut both ways. Rates must never change historical reports by accident — but "what the new prices" is exactly the question the subsidy argument makes people ask. That's a deliberate re-price, not a mutation.
@ryan_davis23 Cost per outcome is the harder number and the right one. The catch is the outcome gets defined outside the tool, a merged PR or a green CI run, so it has to accept a signal from your pipeline rather than try to infer one from spans. On the re-price wrinkle, a deliberate what-if is fine as long as it lives in its own view and never overwrites what last month's report already said.
@asadmalik901 That's the answer I was hoping for — inferring "this span became that merge" from the spans alone is where these usually go wrong.
One thing worth designing early if you do take a signal in: the outcome shows up long after the spans close, sometimes hours, so it's a late join rather than a field written at emit time. Which makes the correlation key the real product decision — something that survives out of the agent, into the branch or the CI job, and comes back attached to the result. The ingest endpoint is the easy half.
And having the outcome defined outside the tool feels like the right place for it. You're the join, not the judge. Congrats on the launch.
@ryan_davis23 The key that survives is the branch name. It's the one thing the agent, the branch and the CI job all already know without anyone writing glue for it, and commit SHA looks better until someone rebases. Late join is the part I'd underestimate though, because it turns span retention into a product decision rather than an infra one. If you only keep raw spans 30 days you've quietly capped how slow an outcome is allowed to be.
The replay functionality could be just as useful for learning as it is for troubleshooting. :)
LangWatch
@ankur_jeswani indeed and to improve your skills for example! Let us know your feedback once you tried, very curious!
LangWatch
@ankur_jeswani the cool thing is that you can even point the agent back at it to troubleshoot itself and improve their skills for next time
congrats on the launch! Does this use any bandwidth during the session? Especially long ones
LangWatch
@mathias_barboza thanks! Nope langwatch doesn’t sit in between
the rate-stamping thread above is the right worry for pricing, but I've got a different edge case: mixed fleets. I run some agent work through Claude Code/API and some through a local model on a Mac mini for anything that doesn't need frontier quality. Does LangWatch have any concept of a zero-cost or self-hosted "model" for sessions that route through a local endpoint, so those sessions still show up with token/call counts in the same dashboard, just without a real dollar figure attached? or does the whole pipeline assume every session maps to a paid API/plan rate somewhere?
Congrats! One feature I'd enjoy is benchmarking cost efficiency across different coding models for the same task. Is that possible?
LangWatch
@himani_sah1 yes, absolutely! This can be done with our experimentation and agent simulations features.
Using our API or MCP, you can grab real cases from tasks you ran through your coding usage and make them as test cases to rerun on different models. Then you can clearly see which ones managed to achieve the goal most successfully, and what was the time and cost to do so
LangWatch
@himani_sah1 Absolutely- you can build you own analytics view for that: sessions running in parallel, cost per session, model, etc! Good luck, and let us know!
Congrats on the launch! Seeing which Claude Code session actually burned the budget instead of guessing would be a relief, and the cache reads split out as their own token class is a thoughtful detail. How would this work for a small team, can we see everyone's sessions in one shared view or is it one workspace per developer?
LangWatch
@doganakbulut both options are available. There is the possibility to work as a team view teams sessions as well as an admin dashboard to keep more control over all