If a user can't disagree with your score, it isn't a measurement
Three of today's launches ship a confident number at the user. How much of this page is AI. How far this repo sits from production ready. How many hours of debt that represents. Different products, same move.
The problem isn't accuracy. It's that nobody can tell you when it's wrong. A linter points at line 40 and you either agree or you don't. A 0-100 score for something unmeasurable can't be argued with, so it never gets corrected, and it trains people to either ignore it or trust it for no reason at all.
That's the failure I keep going on about, output that's plausible enough that nobody complains, wearing a nicer outfit.
The test I've started using before shipping a number: can a user disagree with this and be right in a way I'd have to accept? If not, it's marketing sitting inside the product. Ship the finding that points at a real thing on the screen. The score is optional and usually costs more trust than it earns.
Replies
this matches something I ran into today reading through launches - a call-scoring tool that ties its rubric to deal outcomes, which sounds rigorous until you realize a small team doesn't have enough closed-won/closed-lost data yet for that correlation to mean anything. the score looks scientific because it's tied to a real number (revenue), but if you can't ask "why did this call score 72 instead of 85" and get an answer you could actually contest, it's just a confident-looking guess wearing the outfit of a measurement, like you said. the linter comparison is the right one - a linter tells you exactly which line and why, and you can go look at it. most AI scores skip straight to the verdict and skip the part where you could catch it being wrong.
@galdayan The call-scoring case is worse than most because revenue makes it feel validated. You can be exactly wrong and still correlate, small sample plus a rubric someone wrote by feel, landing on top of a real number. The part I'd steal from linters isn't the line reference, it's that a linter can be switched off. A score a user can't disable when they think it's wrong isn't confident, it's just unfalsifiable.
@asadmalik901 unfalsifiable is the word i was missing there. most scoring tools i've seen don't even expose which inputs moved the number, so you can't argue with it, you can only ignore it. maybe the fix is smaller than a full override - just show the rubric weights behind the score, so a wrong number is at least wrong for a visible reason
@galdayan Showing the weights gets you halfway and then stops. If I see quality is 40% of the score I still can't tell you why this call got a 6 on quality, so the argument just moves down one level and dies there. What I'd want is the input, this sentence at 2:14 is why the number moved. Weights explain the formula, not the verdict.
@asadmalik901 fair, that's the harder ask though - highlighting the actual span means the model has to expose its own reasoning trace, not just its output. most of these tools weren't built to keep that around after scoring. probably why everyone stops at weights, it's the version you can ship without redoing the pipeline
Agreed. AI makes this easy now. Same in my own project: it felt fine at first, then I realized I couldn't explain where the score came from either, and I ended up writing the scoring rules out one by one.
@maxwell_wx Writing the rules out one by one is the fix, and it's also why most people avoid it. Once the rules are explicit somebody can argue with them, and arguing about a weighted list is a much less comfortable conversation than pointing at a model. Worth doing anyway. About half of ours didn't survive being written down.