How do you catch silent data corruption bugs early?
by•
Visual bugs or broken buttons are easy to spot right away, but silent failures are terrifying because everything looks completely normal on the dashboard.
Our scariest silent bug happened when a background worker quietly stopped processing webhook retries due to a silent timeout exception leaving hundreds of user orders stuck in limbo for days without any logs throwing an alert. We ended up setting up an external health ping just to make sure those queues are actually moving every hour. What's your go to way for catching background failures before users notice?
17 views
Replies
I'd alert on the age of the oldest unprocessed order. A worker can keep sending heartbeats while the same orders stay stuck. Include an affected order ID and its last recorded failure in the alert so someone can trace it.
One more way this fails that the oldest-order alert won't catch: the dedupe marker itself.
If you write "event processed" before the work and the work throws, every retry from then on gets skipped as a duplicate. Nothing errors, the queue looks healthy, and the order is stuck for good. I hit this class of bug with payment webhooks, so now the work after the marker has to be safe to re-run, and the marker gets released if the work throws.
The catch is that this only helps if the retry actually shows up. If the sender stops retrying or the worker never picks it up, you're back to needing the age-of-oldest alert from the other comment
A coding agent stubbed a realistic number in a reporting pipeline as a placeholder. It looked plausible, passed checks, and took a day to realize nothing was actually measuring it. The build now fails if any calculation step returns a value without touching raw source rows.