Launching today
Tracely
Production failures become regression tests for AI agents
2 followers
Production failures become regression tests for AI agents
2 followers
Tracely grades every agent trace as it lands, clusters failures into issues, freezes bad runs into hermetic replayable test cases — and blocks the PR that would ship them again. The trace is the test: no hand-authored datasets. Open source, MIT.
Hey Product Hunt 👋
I'm Julien, and I built Tracely because of a loop every agent builder knows: your agent breaks in production, you see it on a dashboard, you nod… and then you sit down to hand-write an eval dataset trying to reproduce what production already handed you for free.
Tracely flips that. The trace is the test:
🔍 Observe — one line of setup: tracely.init(instrument="auto") auto-captures your OpenAI / Anthropic / LangChain / Gemini / Mistral calls with zero span code; a manual API (@observe, agent/tool/llm/retriever/guardrail spans) covers your own logic; and raw OTLP works from any language. The payoff is the read: traces render as conversations — turns → steps → a span waterfall with full I/O on every node. I obsessed over making this the most readable trace view in the category.
⚖️ Detect — this is where Tracely goes deepest. Evaluators are columns on the trace table, not a separate tab, and they grade every run as it lands:
Three levels: conversation, message, or step — and step-level judges can target exactly the span types you care about (tool calls only, generations only, chains, thinking).
Typed outputs, not just a score: define the judge's output schema — booleans, numbers, strings, strict enums the judge cannot escape. A verdict like sycophancy_type: excessive_validation · severity: moderate is queryable data, not prose.
Sequential or batch execution: batch judges run independently; sequential judges receive the previous metrics' results injected as context, so you can chain "did the tool succeed?" → "was the answer faithful to what it returned?". Message-level sequential judges also see the conversation's rolling summary (@HISTORY), built in real time as turns land.
Spend control built in: per-evaluator targeting by agent/env plus deterministic sampling, so the same trace always gets the same judges — and advisory evaluators record a verdict without flipping the PASS/FAIL roll-up while you're still tuning them.
🧩 Triage — failures cluster into issues. 31 broken runs = one issue with a count, not 31 rows.
🧊 Test — one click freezes a failing run into a hermetic case: recorded input, recorded tool/LLM outputs as fixtures, and a fail-to-pass contract.
🚫 Ship — tracely gate replays the suite on every PR and blocks the one that reintroduces the failure. Replay is against fixtures, so it costs $0 in model spend. There's also an adversarial mode: a red-team model improvises against your agent's live endpoint in CI.
It's open source (MIT) — 370+ GitHub stars in its first months — self-host everything with one `docker compose up` (`pip install tracely-ai` for the SDK). Don't want to run infra? There's a hosted cloud with a free plan at tracely-studio.xyz: multi-workspace with team invites, and you bring your own OpenRouter API key for the LLM judges — scoped per workspace, never a shared server key (more providers coming). No key configured? Everything still runs: structural checks keep grading and the LLM judges switch off gracefully.
Two things I'd especially love feedback on: the judge-calibration flow (label judge verdicts against human review, catch an over-flagging judge before it gates a release) and the adversarial scenarios.