Ethan Aspy

Month a partner platform changed their API payload structure without telling anyone turning a regular user ID field into a nested object. That small change broke three days of data across our CRM and email marketing systems.

Of going through thousands of bad records in the database one by one we created a simple validation middleware between our webhook receiver and our internal processing queue. Every incoming payload now goes through a schema check before it reaches any business logic.

If a field does not match the schema we expect the middleware moves the payload to an error log table. Sends a message to our developer channel on Slack right away. This small change has saved us a lot of time that we would have spent fixing data

What error handling strategies do you use to keep your automated workflows safe, from API changes?

When a user is flagged by my safety layer, the app goes quiet for a while. I kept the whole rule in one pure function: it takes the user s recent flags and the current time, and returns whether they re paused, until when, and why. No database, no clock reads inside, and the lengths live in named constants (pause length, repeat window, minimum gap between two flags that count as a repeat).

The rule itself is small, but time rules go wrong at the edges: exactly at a boundary, two flags a minute apart, a timestamp with microseconds and a +00:00 offset. So I wrote 61 tests, then did something I now do for every rule like this: I broke the code on purpose, seven different ways, and checked that a test failed each time. I removed the minimum gap, counted flags I shouldn t count, measured a repeat from the wrong flag, made the pause read fail closed, and so on. All seven were caught. A test that has never failed proves nothing.

Two things the design review found before users did:

1. I first started the soft period (gentler messaging after a pause) at the flag time. That meant it ended the moment the longest pause ended, which is the worst time to switch back to normal. I moved the start to the pause end. That quietly broke something else: the database query only looked back as far as the old rule needed, so a repeat flag would have fallen outside it while the soft period was still running. The look-back had to grow to cover pause plus soft period plus repeat window. A test now fails if I shrink it.

2. Contact-form flags must not count as repeats. The table holds only a user ID, a level, and a time, and I didn t want a text column, so I added a single yes/no column instead.

I run a 3-level classifier (0 normal, 1 distress, 2 crisis) in front of a text generator. At Level 1 or 2 the app skips the generation, shows a support page, and pauses the feature. The final level is the max of three signals: a phrase filter, the classifier, and a level returned by the generation call itself. I tested it across several sets of typed messages: crisis, sarcasm, jokes, other languages.

Then, while testing on production, I tapped one of the three preset answer buttons ( I failed again ) and got the support page and a lockout. At the time, a Level 1 flag meant a 72-hour pause.

The cause: the preset labels went through the classifier exactly like typed text. Run alone, the classifier returned Level 1 on that label in 5 of 5 runs, while the grading call returned 0. Max of signals meant Level 1 won.

Why my evals missed it: every test row was something a person might type. None of them were the strings the app itself sends.