If all your tests pass, how do you know the running system is actually right?
Spent yesterday adding a new API. 54 tests green. No dangling refs. Then I actually called it.
Four things wrong, none of them raised, none logged, every test stayed green.
What broke
Every read-back 404'd because I named the endpoint after the resource instead of copying the path from the spec. An ID pattern used a wildcard the generator doesn't recognize, so it handed callers the literal string ent_%%%%%%%%. Seeded records and runtime records had different ID formats because seeding ran outside the context that tracks them. One endpoint returned a list of lists because a sub-resource got seeded from the envelope instead of the record inside it.
The one that bothered me most
Matched my exact symptom strings at 0.95 confidence. Paraphrased the same bug like a person would type it and got 0.4. It looked deep. It was keyed.
The actual question
How do you catch config bugs like this? Not code bugs. Every layer right on its own, running system still wrong. Green tells you what you meant to build, not what it does.
Launching what this led to tomorrow if you want to see where it ended up.
Replies
three of your four bugs are really the same failure mode wearing different clothes - the endpoint name, the ID format, and the envelope/record shape were all facts that got typed twice by two different code paths (spec vs implementation, seeding vs runtime) instead of coming from one source. that class of bug isn't a testing gap, it's a "there are two copies of the truth and nothing keeps them in sync" gap - tests written against your own hand-typed assumptions will always agree with themselves. the fix that's actually worked for me is generating both the client and the seed/fixture data from the same schema the server validates against, so a drift shows up as a type error before you ever run a test, not as a 404 in prod. the similarity-matching point is a separate and honestly scarier problem though - if your bug-catching tool scores paraphrases lower than exact string matches, it's measuring surface text, not meaning, and that gap will always be widest exactly when someone reports the bug in their own words instead of yours.