Prompt Eval - Stop guessing if your prompt is good. Measure it.
by•
Run your prompt across a dataset of test cases and get a scored report: average score, pass rate, and per-case reasoning that explains every failure. Compare versions to catch regressions. Deterministic, LLM-judge, and reference grading. 20 free credits to start.
Replies
I built Prompt Eval after going through Anthropic's Academy and realizing how much evaluation actually matters. A prompt can look fine on one try and still fall apart the moment you change the input, and most people (me included) never actually test that.
So I tried it on one of my own prompts. It scored 4.9 out of 10, zero percent pass rate. Changed one thing, being clear and direct about the output format, and it jumped to mostly 8 out of 10.
That's the whole idea: write a prompt, test it against real cases, fix what actually breaks, measure it again.
Everyone tests their code. Almost nobody tests their prompts.
Would love your feedback, especially the rough edges.
As soon as people understand how important evaluating system prompts is, their prompt engineering will go to a different level.