Repath
Canary rollouts for prompts & models, gated on quality
Start new thread

Repath - Canary rollouts for prompts & models, gated on quality

by•
Change a prompt or swap a model without guessing. Repath sends a slice of real traffic to the new version, an LLM judge scores both versions on the same criteria, and a controller advances the rollout (5% → 25% → 100%) or rolls it back to 0% on its own, with the worst answers and the judge's reasons logged. Integration is one base-URL change in the SDK you already use: OpenAI, Anthropic, Gemini, OpenRouter or Vercel AI Gateway. Gateway overhead: ~0.4 ms median.

Add a comment

Replies

Best
Maker
📌
Hey Product Hunt 👋 I built Repath because every prompt change I shipped felt like a coin flip. Offline evals passed, and then real users got worse answers and I found out from them. Repath brings the canary rollout to prompts and models: → Point your SDK at Repath (base URL + one header) → Send 5% of real traffic to the new prompt or model → An LLM judge scores both versions on the same criteria, off the request path → Quality holds, it advances: 5% → 25% → 100% → Quality drops, it rolls back to 0% by itself, with the worst answers and the judge's reasons attached Building it taught me a lot. The hardest part wasn't the AI, it was staying out of the way: the gateway sits in your request path, and keeping its overhead under a millisecond (~0.4 ms median) shaped every design decision. Shadow mode, where the candidate runs on every request without users ever seeing it, is next. 7-day free trial, no card. I'd love to hear how you test prompt changes today.