Should an AI agent's cost budget be a hard limit or a prediction?
One thing I've been thinking about while building tooling for AI agents is that a budget and a cost estimate solve two very different problems.
A budget tells you what you're willing to spend. A prediction tells you what a task is likely to cost. Neither guarantees that the task will finish within that amount.
Suppose a coding agent is halfway through a task. It has already spent $0.40, but the remaining work could cost another $0.05 or $2.00. The agent doesn't necessarily know how many retries it will need, whether it will get stuck, or whether it will have to delegate more work.
If we stop it at a fixed threshold, we risk interrupting useful work. If we let it continue because the task is almost done, we risk giving an uncontrolled process unlimited room to spend.
I'm interested in how people are thinking about this distinction.
Would you prefer a strict per-task spending cap, a predicted cost limit that adjusts as the agent makes progress, or a combination of both?
And if you use adaptive limits, what signals would you trust enough to make that decision: historical task costs, the remaining context, tool-call patterns, retry frequency, or something else?
Replies
Count in-flight calls against the account cap too
Before each paid call reserve its maximum allowed cost atomically then settle against actual usage. Otherwise several parallel tasks can each pass the same remaining-budget check and overspend together
@al_kub Exactly. Without atomic reservations, every concurrent task can see the same available balance and make a decision based on money that's already effectively committed elsewhere.
Reserving the maximum allowed cost before execution, then reconciling against actual usage, makes the budget an enforceable constraint rather than just a check.
The tricky part seems to be handling failures cleanly: a worker can time out after the provider has processed the request, leaving you uncertain whether the reserved amount can safely be released. That gets even more interesting when retries are involved.
Would you handle that with reservation leases and idempotent settlement, or keep reservations until the provider's final usage is known?
@kushagra_agrawal8 I would let the lease expire the worker ownership, not release the money
Keep an unknown outcome reserved until reconciled and make settlement idempotent. Retry with the same provider key only if it guarantees deduplication otherwise budget for a second charge
@al_kub That distinction between worker ownership and the reservation itself is important. Expiring the lease shouldn't automatically mean releasing the money when the provider might already have processed the request.
The same-provider-key point is interesting too. Without guaranteed deduplication, a retry isn't just another execution attempt. It's another potential financial liability.
I'd be curious how you'd handle reconciliation when the provider never gives you a definitive answer. Would you keep the reservation indefinitely, or eventually move it into a separate unresolved-liability state that still counts against the account cap?
@kushagra_agrawal8 I would move it to an unresolved state that still counts against the cap, with a review deadline rather than automatic release.
A customer refund can happen separately. Refunding their credits should not erase a possible provider charge from your internal budget.
Both, at different levels. My product isn't an agent, but every song costs me real model money, and I ended up with two layers. The user sees a price per song in credits, which works like your prediction, a known cost per job. Underneath there's a dumb hard cap: one account can start at most 20 songs a day. A paying user never hits it, it exists so a compromised or scripted account can't burn money overnight. I'd do the same for agents: let the adaptive limit make the nice decisions mid task, and keep a hard ceiling that nothing adaptive can raise. The ceiling isn't there for the normal case, it's there for the case your signals didn't predict.
@gbasovv That's a useful distinction, especially the part about keeping the hard ceiling independent of anything adaptive.
The two layers solve different problems: the adaptive limit manages a task's economics while it's running, while the hard ceiling protects against failures the system didn't anticipate.
For agents, I think the interesting question is where that ceiling should apply. Per task, per account, or across all concurrent tasks from the same account? A per-task cap might look safe in isolation but still allow aggregate spend to run away through parallel execution.
How are you handling that boundary in your system when multiple songs are being generated concurrently?
I charge per flashcard generation in a product I build, one credit taken up front with a single conditional UPDATE, and it comes back if the generation fails. Early on every failure refunded. Then I noticed that some failures depend on what the user asked. So the user gets the credit back, asks again, gets it back again, and that's a free loop that never touches the hard cap, because the balance never drops.
So now I split refunds by: could the request's content have caused this? Provider timeout or 5xx, refund with no limit, the user did nothing wrong. Empty answer, too few results, a refusal, all go through a small daily refund cap (3 a day). Past it the user keeps whatever came back and the credit stays spent.
For your timeout question I'd go the other way from keeping it reserved, at least for a user-facing product. The provider may well have billed me, but the user got nothing, so they get the credit back and I eat the cost. With retries in the loop I'd want Aleksandr's idempotent settlement first.
@hamanovich The distinction between infrastructure failures and request-dependent failures makes sense. Otherwise, a refund policy can accidentally become a retry mechanism with no real cost to the user, even when the underlying work keeps failing for the same reason.
I also like the principle that the user shouldn't have to absorb provider failures just because the provider may have charged you. That's a separate decision from how you protect the system against repeated unsuccessful requests.
The tricky case seems to be attribution. An empty response could be a bad request, a model limitation, or just a transient failure. Do you classify these based on explicit provider signals, or do you use the generation outcome itself to decide which refund policy applies?
hard cap per task for sure but i think the more useful signal is progress not spend. if retries keep climbing and the agent is still stuck on the same step, thats the point to pause and ask the human, even if its way under budget. cost tells you how much you burned, not whether its going anywhere. does anyone track spend since the last real step forward instead of total spend
@hitensethiya The spend-since-last-progress signal is interesting because it separates expensive work from unproductive work.
An agent could spend very little while repeatedly failing at the same step, or spend quite a bit while making legitimate progress on a difficult task. A fixed budget alone doesn't distinguish those cases.
I'd want to track progress alongside retries, repeated tool calls, and state changes, rather than treating a lower cost as evidence of healthier execution.
The hard part is defining a meaningful step forward without relying entirely on the agent declaring that it has made progress. What would you use as the signal: a change in task state, a passing test, a changed artifact, or something else?
Hard limit, with the prediction underneath it. We run agents on scheduled marketing jobs, and the failure I actually worry about is not the expensive task, it is the loop that retries quietly at 3am. A prediction cannot catch that, because the loop looks normal call by call. So the cap is fixed and boring, the prediction only decides when to warn a person before the cap hits, and anything that hits the cap stops and says why. Retry count has been the most honest signal for us.
Hard limit, from experience. A few weeks ago my agent burned through a prepaid pack of 50,000 search API calls, mostly on one-off searches I didn't even know about.
Now a hook blocks any ad hoc command that calls a paid API directly. The agent has to ask me first, say what it's for and how many calls, and only then run it. Planned jobs go through a library, so they aren't blocked. On top of that, I watch API usage all the time and get a phone alert when it crosses a threshold.
We hit this with metered voice sessions. We hold the next minute's cost before it runs, so a session can never spend past what's left, but we let it finish the sentence it's in. Hard limit on the money, soft edges on the work.
I gave a coding agent an estimated budget to resolve a broken build. It made the failing test pass by weakening the assertion instead of fixing the code, and the suite stayed green for days.