How do you keep an optional AI call from breaking the main one?
I’m adding a short plan (a small challenge plus two or three steps) to the results page of an AI app. The endpoint that produces the result already chains several model calls: a safety check, the main generation, and two small side calls (a share caption and a memory note), all inside a 60-second function limit. Before adding a fifth call, I had Claude Code read the route and report how close it was to that limit.
The finding was uncomfortable. The two side calls had no timeout of their own. The SDK’s default is one minute with two automatic retries, so one hung side call could run past the 60 seconds, and the platform would kill the whole request before the main result was saved. A code comment even claimed each call had its own timeout. It didn’t.
What I changed: both side calls now get a 10-second timeout and no retries, and their failures are still swallowed as before. The new plan call goes through a helper that makes one attempt with an 8-second timeout and never throws. On a timeout, bad JSON, over-long text or a failed safety check it returns nothing, and the result is saved without a plan. It starts only after every safety gate, at the same moment as the side calls, so it adds little waiting time.
A test reads the route’s source and checks that the plan call sits before the single database insert and that the side calls carry a timeout and zero retries, so a later edit can’t quietly undo it.
The honest weak spot: the worst case is now bounded under 60 seconds, but not by a wide margin on a bad day, because one helper can still make two 10-second attempts. And an optional call that gives up under load means the feature disappears exactly when traffic is highest.
How do you cap latency across chained model calls: a timeout per call, one overall deadline for the request, or both? And how do you decide which calls are allowed to fail silently?
Replies
Both, and a rule for which calls are allowed to fail: anything that only improves a result that already exists. In my song app the song itself is the main call and runs first, because it is the slow, expensive and fallible step. After it, a model listens to the finished track to correct the lyrics, and it runs beside the title and the cover with its own 120 second timeout, best effort: if it fails, the song keeps the planned lyrics and is still delivered. One number that may help your margin problem: that listening call took about 24 seconds with a capped thinking budget and about 80 without, so the model's own thinking setting is worth checking before the retry settings.
I keep the model's part as small as I can and push everything else into plain code, so in my setup anything optional is treated as optional from the start. I also watch API usage all the time and get a phone alert when it crosses a threshold. The test that reads the route so nobody quietly undoes the timeouts is a smart touch.