You can build the feature in an afternoon. The eval is the hard part now.

by•

Building got cheap this year. Judging whether the thing is actually good did not. In 2026 the eval is your spec, and it's the step nearly everyone leaves out.

There's a moment now that didn't exist three years ago. You describe a feature, an agent writes it, and forty minutes later it's running. It works, sort of. It answers, it responds, it does the thing. And you sit there with a genuinely hard question you have no tool for: is it any good?

That question used to answer itself. Old software was deterministic — same input, same output, and you wrote a test that either passed or failed. AI features don't work like that. Ask the same thing twice and you get two different answers, both plausible, one subtly wrong. "It ran without erroring" tells you almost nothing about whether it's right, helpful, or safe. The build got easy. The judging got hard. And the gap between those two is where most maker products quietly fall apart around week three.

The discipline that closes the gap has a name this year: eval-driven development. The idea is boring and it's correct. Before you tune a prompt or swap a model, you write down what "good" means as a set of concrete examples, and you check every change against them. Not vibes. Not "seems better to me." A fixed set of cases with expected behavior that you can rerun. The whole shift in how serious teams build with AI in 2026 is the move away from vibes-based iteration toward something you can actually rerun and compare. Eval platforms are multiplying for a reason — the demand is real.

Here's the part that matters for a solo maker, because most eval writing gets described in language built for a 40-person engineering org and you tune out. You don't need the platform. The minimum viable version fits in a spreadsheet. Collect twenty real inputs — actual things users typed, not inputs you imagined. For each one, write down what a good response looks like, in your own words. When you change a prompt or a model, run all twenty and read the outputs side by side against yesterday's. That's it. That's an eval loop. The magic isn't the tooling; it's that you defined "good" before you started fiddling, so you can't fool yourself into thinking a change helped when it just felt fresh.

Two failure modes to name, because I've hit both. The first is testing on inputs you invented — they're always cleaner and friendlier than what real people send, so you pass your own test and ship something that faceplants on the first weird message. Use real inputs. The second is letting a model grade its own homework. LLM-as-judge is genuinely useful for scale, but if you never read raw outputs yourself, you slowly drift toward whatever the judge rewards, which is not the same as whatever your users need.

I'll be upfront about my bias. At Murror we build emotional AI — a product that helps people journal and understand their own patterns — and evals are hardest exactly where the output is subjective. There's no unit test for "did this reflection feel true to the person who wrote it." A response can be factually fine and emotionally off, and the second one is the one that makes someone close the app. So our evals lean on human reading more than automated scoring, and the "expected answer" is often a feeling we describe as precisely as we can rather than a string we match. It's slower. It's also the only honest way I've found to measure the thing that actually matters. If your product's value is subjective too, don't let anyone talk you into believing a number captures it.

The uncomfortable takeaway: in the era where anyone can build the feature, the eval is the moat. It's the one artifact an agent can't hand you, because it's the encoding of your taste and your knowledge of your users. The makers who win the next stretch won't be the ones who ship fastest. They'll be the ones who can tell, reliably and before their users do, whether what they shipped is good.

Write the twenty cases. Do it today, before the next feature. It's the least glamorous hour you'll spend this week and probably the highest-leverage one.

26 views

Add a comment

Replies

Best

Christian's disagreement-pairs idea is a clever way to find where the eval actually needs to exist, but it only tells you the model is unstable on that input, not which of the two divergent answers is the good one. you still need a human (ideally not the builder) to break the tie. feels like it narrows the blind-reading problem down to a much smaller, cheaper set of cases rather than solving it outright

 Exactly — and that "ideally not the builder" part is the bit I keep relearning the hard way. On subjective output the builder is close to the worst possible judge, because I unconsciously grade toward the version that flatters what I already shipped. The pairs don't remove the human read, they shrink it from "read everything" down to "read the handful where good is actually contested" — which is the difference between an eval you keep using and one you quietly abandon by week two. For the tie-break we've started pulling in someone who didn't write the prompt; even a teammate with zero context beats the builder here.

the teammate-with-zero-context trick is a good one, though i'd guess it has a shelf life too - the third time you pull the same person in for a tie break they've absorbed enough of your taste to stop being neutral. might be worth rotating who does it rather than having one designated fresh reader