I’ve been building an open-source tool called DecGuard around a problem I kept coming back to: testing probabilistic decision systems is a bit different from evaluating general-purpose LLM output. If a model is supposed to return something structured like a choice, a boolean decision, or an ordered score with probabilities, there are some fairly concrete things you probably want to know: does reordering the options change the answer? does harmless formatting change the distribution? did calibration get worse after changing the model/backend? can I reproduce a failure later? can CI reject a version that violates one of these assumptions? what happens to reliability on production records or specific segments? DecGuard puts those things into a small decision contract and uses the same contract for normal tests, metamorphic fuzzing, regression comparisons, replay, production checks, and reliability gates. It currently supports Choice, Noul/boolean and Score decisions, and the backend layer is deliberately interchangeable. I’ve tested the System One integration end-to-end with real Kev and Jev backends as well. It’s still an early version, so I’m mostly interested in whether the abstraction itself makes sense to people building this kind of system, and where it breaks down in real use.

DecGuard

Reliability testing for probabilistic AI decisions.

Reliability testing for probabilistic AI decisions.