Launching today
TraceLLM
OpenTelemetry for production AI applications
104 followers
OpenTelemetry for production AI applications
104 followers
Tracellm is an observability platform for production AI applications. Monitor prompt execution, token consumption, latency, spans, errors, and model calls across your LLM workflows. Export traces using OpenTelemetry (OTLP) and quickly identify bottlenecks before they impact users
TraceLLM
Every question on that list is about a failure that announces itself. Failed request, slow call, token spike, all visible.
The one that costs most in production is the request that succeeded. No error, normal latency, sensible token count, and the answer was confidently wrong. To catch that afterwards the trace has to hold what the turn was grounded in, which retrieval actually landed and what the tool returned, attributed to that turn. Otherwise the worst incident of the year looks like a healthy span.
Does a span carry the retrieved content and tool results, or the call and the timing only?
TraceLLM
@jernej_jan_kocica yes, it can carry the grounding context and tool results, not just call timing, but we make that explicit and configurable because production AI traces can contain sensitive customer data.
The thing I keep wanting from LLM tracing is MCP tool calls as first class spans, not just the model call with a blob of tool JSON hanging off it. Half my latency is the tools, not the model. Does the OTLP export break tool calls out as child spans?
TraceLLM
@dalemooney Yes. TraceLLM has a first-class tool span kind, so tool calls do not have to live as a JSON blob on the model span.
In a typical agent/RAG flow you can emit separate spans for agent.plan, tool.crm.lookup, retrieval.docs.search, and openai.chat.complete. Each one gets its own timing, status, metadata, optional input/output, and errors.
On OTLP export, those TraceLLM spans are mapped to real OpenTelemetry spans, so in SigNoz/Tempo/Honeycomb the tool latency can be inspected separately from the model latency.
The current caveat is that this is SDK-instrumented today: you create the tool spans from your app code. Automatic MCP tool-call capture is on the roadmap, so MCP calls can become first-class spans without as much manual wiring.
@jyotishmoy_deka Mapping to real OTLP spans rather than a proprietary shape is the answer that matters, because it means I keep whatever backend I already have. Good.
On the automatic MCP capture, one thing worth deciding early. The interesting boundary isn't inside the client app, it's the MCP call itself. An MCP server is usually a separate process, and often a separate deployment entirely, so the spans that explain a slow tool call live on the far side of that hop. If you instrument only the client, tool.crm.lookup becomes one opaque span with the network time, the server's own work and whatever it called baked into a single number, which is the same blob problem one level down.
So: do you plan to propagate trace context across the MCP boundary, so a server that emits its own spans joins the same trace rather than starting a new one? If the answer is yes, the thing to nail down is where the context rides, since the protocol has no obvious header slot for traceparent the way HTTP does.
Either way it's the difference between telling me the tool was slow and telling me why it was slow, and only one of those is worth wiring up.
TraceLLM
@dalemooney That's a great point. Today TraceLLM can make the client-side MCP/tool call a first-class tool span, so you can see that tool.crm.lookup took 800ms instead of hiding it inside the model span.
But you're right that this only explains the outer call. If the MCP server itself does database/API work, the useful next step is propagating trace context across the MCP boundary so spans emitted inside the MCP server join the same trace.
That's the direction we want for MCP support: client tool span first, then server-side MCP spans attached to the same trace, so you can see not only that a tool was slow, but why it was slow.
Pazi
This looks super useful — OpenTelemetry-based observability for LLM apps is a great idea, especially catching bottlenecks early. Congrats on shipping!
TraceLLM