Hi Product Hunters! 👋 Evaluating RAG pipelines and LLM agents in production is currently a nightmare. Most frameworks act as "black boxes" and average out metrics indiscriminately. Worse, they allow the Judge LLM to have a temperature > 0.0, turning your evaluation into a stochastic random number generator (the "Confident Idiot" problem). I built TRI·TFM v3.0 Framework to fix this. Over the last few weeks, we ran massive stress tests across multiple domains. We published our research logs, CSVs, and Python codebase to prove two things: 1. You must lock your evaluator's temperature (T=0.0) to achieve zero-variance evaluation. 2. A dynamic weighting formula (Bal = 0.75F - 0.25B) is the only mathematical equilibrium that prevents "Metric Gaming" (where models generate superficial water text to avoid bias penalties). The repo includes the full evaluation pipeline, config scripts, and our research protocols. I’d love for the AI engineering community to tear our methodology apart, test it on your own RAG pipelines, and let me know how it performs!

TRI·TFM v3.0 Framework

Deterministic, LLM-as-a-Judge evaluation framework.

Deterministic, LLM-as-a-Judge evaluation framework.