Hey Product Hunt!

I built sandtable to fix a specific gap in agent testing: a carrier-sales agent says "Booked L-1001 at $2,100." The call transcript sounds flawless, so an LLM judge model passes it. But the carrier's insurance had expired, a fact living in a database record, not in the dialogue.

sandtable tests conversational agents against stateful, simulated environments to catch these hidden failures before production.

How it works

Simulated callers interact with your agent in real time.
Stateful mocks capture every tool call and evaluate business rules before execution.
Deterministic grading looks at the actual DB records and rules broken. No LLM judge variance—the same transcript yields the same pass/fail result every single time, evaluated over $pass^k$.
Repo-native setups define your domain via simple YAML (generated from your OpenAPI spec, SOPs, and transcripts) that runs in CI on every commit.

Key distinctions

vs. Harbor: Designed specifically for real-time conversational business logic, rather than containerized coding tasks.
vs. LangWatch, DeepEval & IntellAgent: Eliminates subjective LLM judges. Failures pinpoint the precise broken rule and database record.
vs. τ-bench: Adapts the benchmark's underlying methodology into a practical dev tool for custom domains rather than fixed static datasets.
vs. Hosted Voice Platforms: Open-source, runs locally, imports Vapi, Retell, and ElevenLabs agents, and can grade those platforms' test calls.

Current limitations: Text-only (no audio/latency benchmarking yet), no LLM tone scoring, and local file reports instead of a hosted dashboard. This is all work in progress!

Try it in 10 seconds (no API key needed): npx sandtable-ai demo

(MIT-licensed on npm and PyPI, with a Claude Code plugin.)