This is the 2nd launch from OpenObserve. View more
AI Observability by OpenObserve
Launched this week
OpenTelemetry-native observability for agents and LLMs
Your agent cost $40 and took 34 seconds. But why? OpenObserve traces every agent session across models, tools, services, datastores, and user sessions so you can see exactly where time, money, and quality went. Detect loops, run online evals, and follow failures from the LLM call through your backend and database, alongside the logs, traces, and metrics from the rest of your production stack.
rust-based architecture likely plays a big role — the difference in memory and resource usage compared to other alternatives is remarkable, setup for clients and server is also easier
Hi Emilio, thank you for the 5-star review, Emilio! It’s awesome to hear that you found the setup easy and the resource usage remarkable compared to ElasticSearch. Point taken on the mobile UI—we're working hard to polish that up soon!
Thank you @Walter Duan, Your support is the driving force behind our continuous improvement, and we will continue to provide everyone with better products.
Is the trace approach applicable to multiple agents, or is it designed for single agent flows only? Btw, Congratulations @hengfei_yang & team @OpenObserve ✌️🎊.
@aymi_malik Thank you! It’s designed to work with multi-agent flows as well, not just single-agent setups. You can trace interactions across agents and follow the flow end to end, which makes it easier to understand where latency, errors, or unexpected behavior are coming from.
@aymi_malik Thanks! Multi-agent is the case it was built for — single-agent is just the simple end of it.
Agent name and id are first-class dimensions, so every span knows which agent produced it. Sub-agents show up as branches of the same session rather than separate runs, the agent graph shows how agents, tools and models connect, and you can put agents side by side — cost per session, steps, tool failure rate.
What are you running — one framework, or agents spread across a few services?
Report
@hengfei_yang for someone who wants to configure evaluation pipeline, how easy is it going to be? I am also looking to score the responses, say hallucination, risky and etc. With the agentic workflows we have lots of tool calls and that takes a while so how could we trace these tool calls and improve them?
You can set up an evaluation pipeline without building a separate evaluation data pipeline. OpenObserve’s Online Evaluations lets you define scorers using an LLM-as-a-judge or a remote scoring endpoint, and apply them at the span, trace, or session level. So you could have separate scores for things like hallucination, safety/risk, quality, or whatever signals matter for your application.
@chaitanya_sistla For the tool-call side, the OpenTelemetry instrumentation captures tool invocations as spans alongside LLM calls, agent steps, and retrieval, so you can open the trace and see exactly where time is being spent and which tool calls are contributing to latency or failures.
You can then use those traces as evaluation cases too ,traces can be promoted into datasets, and experiments let you run the same cases against different versions and compare whether things actually improved.
Evals — you pick a judge provider once, then switch scorers on. Groundedness/hallucination, safety, relevance and tool effectiveness ship built-in, and they run continuously against production traffic rather than only offline datasets, so you catch drift instead of just errors. Custom means a prompt (LLM judge) or a small code scorer.
Tool calls — every call is its own span with latency and result, so inside a session you get tool hotspots: which tool is slow, how often it's called, how often it fails. Across sessions you can rank tools by total time spent. That's usually where the win hides — one tool called five times when once would have done.
Which tools hurt most for you, retrieval or external APIs?
Report
How long does it take to go from a fresh install to seeing my first agent trace? What does the instrumentation actually look like?
@charity_lee1 Fresh install to first trace is a couple of minutes.
If you're monitoring a coding agent like Claude Code or Codex, instrumentation is literally a few environment variables — endpoint, auth header, exporter flags. No code.
If it's your own app: already emitting OpenTelemetry? Point the OTLP endpoint at OpenObserve and you're done — the LLM and agent spans arrive through the pipeline you already have. If not, you add the auto-instrumentation for your framework, usually an import and one init call.
Which is your case — a coding agent, or your own app?
@charity_lee1 Adding to Hengfei's answer, the "couple of minutes" is real, and the reason is that we didn't invent a new agent, SDK, or data format. It's OpenTelemetry end to end. The same OTLP endpoint that takes your logs and traces takes your agent spans.
What you should look at first is the Agent Graph. Every LLM call, tool call, and handoff in the session laid out as a trace, with tokens and latency on each step. That's usually the moment people realize how much their agent was doing that they couldn't see.
If you want, tell me your framework and I'll point you at the exact init snippet.
Report
Can I see what AI agents are doing when they interact with each other, and trace where things go wrong if one of them starts going off the rails?
The agent graph gives you the shape: which agent called which, where the tools and models sit. The session view gives you the sequence — turn by turn, with the input and output captured on each span, so you can see exactly what one agent handed to the next. That handoff is usually where things start going off the rails: the second agent got a truncated or badly worded brief, and everything after it is downstream of that one moment.
Repeats get flagged even when nothing errors, which is the nasty case — no exception, no timeout, just the same step eight times.
What does off the rails look like in your setup — wrong tool, or wrong plan?
Report
@hengfei_yang Both, if they’re internal. What's tricky is when our agent interacts with other external agents we can’t observe directly. Can we trace what our agent sent, what came back, and how it reacted, so we can test and improve our own agent to produce the correct response, especially when that response is customer-facing?
@sara_dph Yes — and the way to think about it is that the boundary is what matters, not their internals.
Your agent's call to the external one is your span: what you sent, what came back, status and latency, sitting in the same session as everything your agent did next. So the response and your agent's reaction to it are side by side, which is the part you can actually fix.
Then the loop: sessions where the external response was odd get flagged into a review queue, become a dataset of responses you actually observed, and you re-run prompt changes against that set before shipping. Since the output is customer-facing, you can also score the final answer continuously in production.
One caveat — external agents drift. Capture their real responses as your test cases rather than assuming a stable contract.
Are those external agents over HTTP/MCP, or something more custom?
@sara_dph Yes , You can trace agent-to-agent interactions, tool calls, and LLM calls in the same trace, so when one agent starts going off the rails, you can drill into the span hierarchy to see where the behavior changed.
It’s OpenTelemetry-native, so you don’t need a separate telemetry pipeline. We have integrations for frameworks like OpenAI Agents, AutoGen, CrewAI, LangChain, and more.
Report
Does ai observability share the same storage and query layer as the APM/Infra monitoring engine?
@vijeeta_mittal1 it does happens automatically, we have something called auto-discovery which detect what services are sending data and what dimensions can be accounted to correlate it.
What instrumentation do you use? something proprietary or standard like otel?
@vijeeta_mittal1 Yep, and that’s actually one of the things I find most useful about this. AI telemetry isn’t sitting in a separate data layer, so you can correlate it with your existing application and infrastructure telemetry in the same place.
Are you currently running your AI workloads alongside your existing APM/infra stack, or are you using a separate tool for AI observability?
Report
@simran_kumari11 We’re using a separate tool for AI observability today. If everything is in the same data layer, can I actually correlate an AI session with the underlying application traces?
@simran_kumari11@vijeeta_mittal1 yes correlation works across logs, metrics, traces, RUM etc. So you can very well check underlying traces from AI session.
@vijeeta_mittal1@ashish_kolhe2 Yeah truee! The idea is that you don’t have to jump between systems to understand what happened. You can follow the AI session into the application and infrastructure telemetry and see where the actual issue occurred.
We have heard of cases where the AI call looked fine but something downstream was actually causing the problem? Have you run into one @vijeeta_mittal1 ?
@vijeeta_mittal1 okay, we are 100% Otel native to be precise with O2, teams can achieve a complete no vendor lock in with nothing proprietary on the collection as well as storage side.
Report
@manas_sharma9 That's good to know, can you share any guides you might have to get my claude code usage data into your platform?
We've been building AI Observability into OpenObserve for the last few months, and it's finally live.
Everyone is shipping agents, but most tools only tell you the request went through, not whether the answer was any good. a hallucination still returns 200 OK. that's the gap we're closing.
how?
- trace every agent, tool call and model request
- score quality on live traffic, and run experiments before you ship
- send weak traces to a human, then turn them into eval datasets
More importantly it's one unified platform. LLM traces sit next to your logs and infra, OpenTelemetry native, self host or cloud, priced per GB not per span.
It's in beta and we'd love practitioner feedback. if you're running agents in prod, what's painful for you today?
Hi Product Hunt, Simran here from the OpenObserve team.
One thing we kept running into while working with AI workloads: an LLM call can succeed while the agent still fails.
That’s the gap we wanted to solve with AI Observability.
Instead of looking at LLM calls in isolation, we wanted to see the entire session: LLM calls, tool calls, tokens, cost, latency, evaluations, and the infrastructure underneath it.
And because it’s built into OpenObserve, those AI traces live alongside your existing logs, metrics, and application traces. No second observability stack just for your AI workloads.
It’s also fully OpenTelemetry-native, so you can bring telemetry from the frameworks and instrumentation you’re already using.
If you’re already running AI workloads in production, I’d be curious to hear what you’re actually using today to debug them.
Report
@simran_kumari11 Thats great! I already use OTEL for Logs and Traces. Do I need any changes in current setup to start with AI Traces in openobserve ?
@omkark If you’re already using OTEL for logs and traces, you’re pretty much halfway there. You can send the AI telemetry through the same OTEL pipeline, so you don’t need to set up a separate observability stack.
OpenObserve
Hi Emilio, thank you for the 5-star review, Emilio! It’s awesome to hear that you found the setup easy and the resource usage remarkable compared to ElasticSearch. Point taken on the mobile UI—we're working hard to polish that up soon!