What was your worst "forgot to clean up temp files" production disaster?

by•

We pushed a simple media compression service thinking it was harmless but completely forgot to add a routine cleanup for the processing cache.

Within 48 hours hidden temp files quietly swallowed every gigabyte of disk space, crashed the Docker daemon and took our entire staging setup down.

It's amazing how fast a tiny oversight turns into a full blown infrastructure emergency.

What is the most embarrassing resource leak you've accidentally deployed?

25 views

Add a comment

Replies

Best

Not disk, but the same family: test data left lying in production. My app's store screenshots are taken from 44 seeded demo accounts, one per language, and each has a fake music release in the database so the release screen looks real. Those fake releases looked submitted, so an hourly sync job asked the distributor's API for their status, three tries each, with an id the distributor never issued. Dozens of 401s in a row, after which the distributor refused even our valid token for a while, and our own watcher paused releases for everyone, twice, before the cause was found. The fix is one flag on every fixture that the sync skips, and a rule: never call an outside API with an id it did not issue.

@gbasovv That distinction between fixtures and real IDs is a great guardrail. Marking fixtures at the source makes the sync rule much harder to bypass accidentally.

The 48-hour timeline is a good reminder that “temporary” still needs an owner. A TTL, a disk cap, and an alert before the filesystem fills would catch this class of issue early.

Ours was a tiny one. Rendered files expire after a set retention, but the zips of batch jobs were written without an expiry date, so each zip stayed in storage until its whole user workspace was deleted.