Launching today
telemetry.dev
LLM observability for AI agents
292 followers
LLM observability for AI agents
292 followers
telemetry.dev shows each run of your AI agent as one trace. Each model call and tool call is a span with its duration, tokens, cost, and error. Prompts and evals use the same traces: change a prompt without a deploy, then run it on a dataset and compare the scores with production. Alerts tell you about errors, cost spikes, and agent loops. Connect with our TypeScript or Python SDK, or with the OpenTelemetry exporter that you already use.
Free Options
Launch Team
CloudflareDeploy serverless functions globally in seconds
Promoted
Hi Product Hunt, I'm Ephraim, one of the makers of telemetry.dev.
When an agent breaks in production, you want to know which step did it and what the run cost. telemetry.dev shows each run as one trace, with every model call and tool call as a span with its tokens, cost, and error.
New in this launch:
- Prompt management: change prompts without a deploy
- Evals: test a new prompt version on a dataset before it goes live
- Alerts when errors pile up or a run gets expensive
- A CLI and an MCP server, so your coding agent can read your traces
It's built on OpenTelemetry, with TypeScript and Python SDKs for OpenAI, Anthropic, the Vercel AI SDK, LangChain and more. The free plan needs no card, and the code PHBU7A47FH gets you 1 month of Pro.
What's the hardest part of debugging your agents right now?
absolute fan of @ephraimduncan's work. the maker of @blocks.so keeps cooking!
@fmerian thank you for the support🫡❤️
Congrats Ephraim. We run a remote MCP server, so the agent loop lives in the user’s Claude, not on our side. All we see are separate tool calls coming in. Can telemetry.dev group the calls from one conversation into a single trace from the server side, or does it have to sit inside the agent?
@skalmurad Thanks! Short answer: not from the server alone, at least with claude.ai today.
To group calls by conversation, your server has to know which conversation each call came from. Claude doesn't tell it. Each tool call arrives with no conversation ID and no trace ID, so two calls from the same chat look the same as two calls from different chats.
It works when the client sends that information. Claude's API MCP connector sends a trace ID with each call, and our MCP package picks it up automatically. claude.ai and Claude Desktop don't send it yet.
One workaround: your tools can return a session ID that Claude passes back on later calls. It works most of the time, but it depends on the model remembering to do it.
What you get today from the server side: add @telemetry-dev/mcp to your server, and you see each tool's calls, failure rate, latency, and the reasons calls failed.
Impressive tracing. Questions: does pricing scale for high-volume inference? Self-hosting option? How does this compare to vanilla OpenTelemetry sampling?
@omri_ben_shoham1 Thank you❤️ Pricing is per event, not per token, Team is $50/month for 2M, with no overage charges (yet - we should have it soon). We don't have self-hosting also and we don't sample on our side, so use any OpenTelemetry sampler you like.
The MCP server part is what caught my eye — when a coding agent reads a trace to fix a failing run, does it see the full prompt and tool inputs, or can you redact sensitive parts first? I'm wondering how you keep debugging useful without leaking things like API keys or user data into the agent's context.
@jcbuilds Thanks you, by default the agent doesn't see prompts or tool inputs. It sees the steps, timing, tokens, cost, and errors. Full payloads need a separate permission that you turn on when you connect the agent.
We also redact API keys, tokens, passwords, emails, phone numbers, and card numbers before we store anything, so the agent never sees them. You can add your own patterns, or mask data in the SDK so it never leaves your app.
ive built ai apps and always felt I needed something like this to debug and improve my ai responses. dope launch
@soham_bharambe thank you, let me know if you need any help onboarding
Congrats on the launch! I love how each run shows up as one trace — is cost or errors usually the first thing you check in a trace?