One credit should never equal one generation. That was my mistake.
When we set up credits, one credit meant one generation. Easy to explain, and wrong.
A generation isn't one thing. The same request routed to a cheap model versus an expensive one can differ by an order of magnitude in what it actually costs us, and a retry after a bad output costs full price again. So a flat credit means the people doing simple work subsidise the heavy users, and the users you most want, the ones pushing the tool hard, are the ones losing you money.
The fix wasn't a better price. It was making a credit reflect real cost and showing that in the UI before someone commits. People are fine paying four credits for the better model. They're not fine finding out afterwards.
The part I still haven't solved is retries. If our output is wrong, charging again is a tax on our own failure. If we don't charge, we eat the compute on every ambiguous case, and "was that actually wrong" is not a question you want to arbitrate at scale. Right now we refund obvious failures and swallow the rest, which won't hold.
How are you handling retry billing? I haven't seen anyone talk about it honestly.
Replies
This is a good lesson. Flat credits feel simple at first, but costs can vary a lot depending on model choice, context size, and output length. Showing an estimated cost before generation and charging based on actual usage feels much more transparent.
@alpertayfurr Estimate before generation we shipped, and it moved trust more than the price change did. The part I can't crack is still the retry, because usage based billing means our own bad output lands on the customer's invoice, and every answer I've been given is either refund on request, which is a support queue in disguise, or don't refund, which is the thing people cancel over. Nobody seems to have a rule that works without a human reading the output.
@asadmalik901 I’d solve this at the billing layer: charge per completed job, not per model attempt. One generation + one retry would be included in the quoted price, while API failures or outputs that fail validation would never be billed. Any further retry starts a new job.
That way you price the average retry cost into the job itself instead of making the customer pay for your model’s failure.
the split I'd try isn't refund vs don't, it's classifying WHY the retry happened. a schema violation or a truncated/malformed output is on you, that should always be free since it's a system failure, not a quality judgment call. a subjective miss like wrong tone or not quite what they meant is genuinely ambiguous and charging for it is defensible if you're upfront about it. right now most tools lump both into one bucket, which is probably why every answer you've gotten feels unsatisfying - you're trying to find one rule for two different problems.
As you’re well aware, non-developer customers don’t care what a token is in the first place, and if you try to explain it, they find it annoying. In my experience, at least — when you’re using a model via API, the pricing feels clear from a developer’s perspective, but once you bake it into a product, by the time it reaches the actual end user, tokens act like a thick fog that makes the origin and basis of the charge hard to see. That’s why I personally think attaching an easy-to-grasp unit like credits is the right direction.
Retries fall under the same logic. Telling a customer “well, internally there are tokens firing off all over the place, so retries cost money too, haha” isn’t going to land — because what they want is a promise of the result they asked for. It’s a genuinely hard problem, but I think pricing with retries already factored in is the basic move, and after that it comes down to writing the actual policy and working through whatever else fits your specific product.
@omri_ben_shoham1 Splitting it by cause is the answer I was missing. Schema violations and truncated output I can detect in code with no human in the loop, and that's most of what we refund by hand today. The tone misses stay ambiguous, but at least I can stop trying to write one policy for two different failures.
@alpertayfurr Quoting one retry into the job price is the cleanest version of this, and it works if the retry rate is stable across jobs. Ours isn't. The long-form work fails validation far more often than the short stuff, so a single blended quote either overcharges the easy jobs or eats the margin on the hard ones, and quoting per job type starts to look like the complexity I was trying to escape.
@yonghahwang Credits as the unit I agree with, that's not the part I'd change. Where I'd push back is on the fog, because the fix wasn't hiding the cost better, it was putting the credit price per model on screen before the click. Same price, far fewer complaints. Nobody needs to know what a token is, they need the number before they commit instead of after.