QuorumRouter
Fail-closed routing and safe execution for AI agents
17 followers
Fail-closed routing and safe execution for AI agents
17 followers
QuorumRouter is a Open source Deno framework with two distinct multi-model paths. Best Route fans prompts out to independent model adapters, validates every response with Zod, requires quorum, and synthesizes the strongest answer. Agent Chat gives different models shared conversation context so they can challenge, revise, and converge across visible turns. Any repository mutation still requires external SafeLoop authority, exact-digest approval, watched execution, and verified artifacts.
the "return conflicting digests for manual review instead of auto-picking a winner" choice is the right instinct - a system that quietly resolves disagreement is more dangerous than one that admits it can't decide. one thing I'd want over time: if the same adapter is consistently the one whose digest doesn't match quorum on certain task types (not randomly, but a pattern), does QuorumRouter surface that as a signal that one provider is systematically weak there, or does every disagreement look like an isolated one-off unless you're tracking it yourself?
@galdayan Good point. Today, each disagreement is preserved in the trace, but we don’t yet aggregate patterns by adapter and task type. That’s a natural next step: surfacing repeated outliers as a reliability signal without treating minority answers as automatically wrong.
@sakamoto_sann that's the right framing - a per-adapter reliability signal over time is way more useful than treating each disagreement as independent noise. keep the "minority isn't automatically wrong" part though, that's the detail that stops this from just becoming majority-rules with extra steps
@galdayan Exactly. Reliability should be an advisory signal, not a shortcut to weighted majority rule. Repeated outliers can trigger inspection, but the dissent must stay visible—and without ground truth, the minority shouldn’t be labeled wrong. We’ve shipped per-run Decision Reports; task-specific calibration is the next layer.
@sakamoto_sann per-run Decision Reports as the layer above the aggregate signal makes sense - keeps a human or downstream process able to actually inspect why a given run got flagged rather than trusting a single reliability score blindly. curious how granular task-specific calibration ends up being, per task type or closer to per prompt pattern
@sakamoto_sann that makes sense, caller-defined pattern labels means the taxonomy quality is on us not the tool. what happens when a caller mislabels a pattern bucket though - does the parent fallback catch that kind of drift automatically or would a bad label just quietly sit there until someone notices the calibration looks off
How does the quorum threshold get set, and can I tune how strict the validation is per model adapter?
@satemireono The quorum is set with minSuccessfulAdapters. By default, QuorumRouter requires 2 valid adapter outputs, or 1 when only one adapter is configured. If adaptive routing selects a smaller eligible set, the effective threshold is capped to that set.
Zod validation is currently strict at the shared contract level: every adapter must return a valid ModelOutput, and the synthesis must match FinalSynthesis. An adapter can add stricter provider-specific validation internally, but there isn’t a declarative per-adapter “strictness” setting yet. That’s a useful addition we’re considering.
This accurately distinguishes:
- configurable quorum count;
- shared schema validation;
- adapter-specific custom validation;
- the absence of a first-class strictness knob.
How does quorum validation actually pick the strongest answer when models disagree, and is there a way to see why one response won over another?
@lknurbykgesmo2 Quorum validation doesn’t simply crown the majority answer. Each response must pass the same schema, then the synthesis step compares the valid candidates. The trace shows which models responded, what passed validation, and the final selection path. We’re working on making the “why this answer won” evidence even clearer.
This one is close to home, I also run multiple models against the same question in my product. The interesting case is when they genuinely disagree. Quorum works when there is a clear majority, but sometimes the minority answer turns out to be the right one. How does Best Route handle a real split, does it synthesize anyway or surface the disagreement to the caller?
@henry_s_jung A real split is preserved, not quietly erased. Best Route can synthesize valid candidates, but conflicting digests remain visible in the trace; it doesn’t pretend disagreement equals certainty. For higher-risk workflows, that split can be surfaced for manual review.
What happens when the models don't reach quorum — do you return an error, retry, or fall back to the strongest single answer? That edge case seems like where the fail-closed promise actually gets tested.
@andrei_rebrov1 If the required quorum isn’t reached, QuorumRouter fails closed with a quorum error. It does not silently fall back to the strongest single answer. Retries only apply to bounded transport failures, not semantic disagreement or validation failure.
Curious how it handles costs in Best Route when multiple models get pinged per request, any built-in controls or do we wire that up ourselves?
@necdet654042 The core library supports estimated per-adapter costs and a budget manager that can block candidates before invocation. The generated Best Route demo doesn’t yet include live token billing or a CLI spending cap, so production users should provide cost estimates and bound the candidate set.