My own test suite took my AI app down. What are you doing about free-tier daily limits?
I’ve been testing a safety check for an AI app by running a script of test messages many times against the model. Every run was fine until it wasn’t: after a day of repeated runs, calls started failing with 429 errors, and waiting 60 seconds between retries stopped helping.
I assumed the limit was per minute. It wasn’t. On the free plan for the model I use, the limits are 30 requests a minute, 1,000 requests a day, 8,000 tokens a minute and 200,000 tokens a day. I believe I hit the daily cap, because the app itself also stopped generating responses at the same time. My tests and the app share one account, so they share one budget.
What I’m changing: run tests in small batches, never repeat full runs, and plan a backup provider so a quota problem doesn’t become an outage.
How do you keep testing from eating your production quota? Separate accounts, a cheaper model for tests, or something else?
Replies
Separate keys for tests and prod is the first fix, so a test run can never take the app down again. After that, most of my test runs don't hit a paid or rate-limited API at all. I record real model responses once and replay them, so the logic around the model gets tested for free. For things that need a live model, I run a small local model with Ollama. It's worse than the real one, but good enough to catch broken prompts and parsing. The real model only runs a small final batch before shipping.
Two things that keep this from happening to me. First, the test that spends money is marked as such and skipped by default. The end to end suite has one scenario that runs a real generation, and it only runs when I pass an explicit flag; everything else drives the app without touching the model. Nobody runs the paid one by accident, the coding agent included. Second, a per account daily cap on the backend: twenty songs a day, checked and incremented in one transaction so two taps that arrive together cannot both pass. It is a spend guard, not a product rule, a paying user never meets it since a week of credits is twenty songs anyway. It exists for exactly your case, a script or a compromised account burning through the budget. And I never built on a free tier for the model calls. A paid project has quotas sized for an app, and the bill, not a 429 at the worst moment, is what tells me I am overdoing it.