ConvoProbe lets you design multi-turn conversation scenarios and run them against your Dify chatbot automatically to measure response quality. Existing eval tools (LangSmith, Langfuse, Opik) work great for tracing and single-turn evaluation — but they don't support designing and executing multi-turn conversation scenarios end-to-end. ConvoProbe fills that gap.

Framer — Launch websites with enterprise needs at startup speeds.

Launch websites with enterprise needs at startup speeds.

Hi everyone! I built ConvoProbe to solve a problem I kept running into with my own Dify chatbots. Single-turn testing always looked fine. But once real users started having 3-4 turn conversations, quality would drop — context loss, mixed-up information, regressions after workflow updates. The worst part: these failures are silent. No errors in the logs, just plausible-sounding wrong answers. I looked into existing tools (LangSmith, Langfuse, Opik, etc.) and they're great for tracing and single-turn evaluation, but none of them support designing multi-turn conversation scenarios and running them against a Dify chatbot. So I built one. It started as an internal tool but turned out to be useful enough to publish. Would love to hear how others are handling chatbot quality — it feels like an underexplored problem.

Framer — Launch websites with enterprise needs at startup speeds.

Launch websites with enterprise needs at startup speeds.

ConvoProbe

Automated scenario testing for Dify chatbots

Automated scenario testing for Dify chatbots