The Bug That Only Appeared After I Deployed

by•

I tested locally for two days. Everything worked. I deployed, and within an hour three users reported the same issue.

The difference was environment variables. One flag was set in development and not in production, and the code had a silent fallback that made it look like it worked when it didn't.

I had tested the code. I hadn't tested the configuration. Now I keep a checklist of every environment-specific value and verify each one after deploy, not before.

What's the bug that only showed up in production and taught you to check the boring stuff?

6 views

Add a comment

Replies

Best

Mine was a deploy that was fine and still lost work. My backend runs on Firebase, and a freshly created trigger drops events for its first minute or two: a document written 45 seconds after the deploy was never picked up, one written five minutes later was. Locally there is no such window, so no test could have shown it. Now any new trigger gets a few minutes before anyone tests it, and a document that got stuck is recreated rather than debugged. On your silent fallback, the rule I keep is that every branch that quietly gives up writes a log line saying so. A fallback that hides the missing flag is the expensive part, the flag itself is cheap.