Launching today
CAFE — Compound-AI Factorial Evaluation
Stop guessing which AI config is better. Prove it.
7 followers
Stop guessing which AI config is better. Prove it.
7 followers
CAFE treats every knob in your AI pipeline - retrieval, reranking, prompts, models, and tools - as an experimental factor. It runs factorial experiments, evaluates outputs using a configurable LLM (and optionally human reviewers), and applies mixed-effects models to determine: - Which techniques actually improve quality - How much each technique contributes - Whether the observed differences are statistically significant Open source and self-hostable.
Would love to see automatic cost and latency tracking baked into each experimental factor so you can weigh quality gains against compute spend, not just statistical significance. Right now it’s all about the output score, but in production the more expensive combo that wins by 2% isn’t always the right call.
@douakrimdriss Hi, totally agree! Latency, tokens and cost can already be tracked and analysed (see pareto() function). However, I just realised that we do not yet attribute it to each factor the way we do for quality, but rather to the configs as a whole.. That's a genuinely good extension, adding it to the roadmap. Thanks for the thoughtful comment!
finally a tool that says "statistical significance" and means it. love that you can swap the eval LLM and rerun everything without rebuilding the whole experiment.
@nurayg61521 Thanks for checking it out :)