Your model provider can change your product overnight and you find out from a user
We run 30+ models across six studios in Arteza and every one of them belongs to someone else. If a provider retires a version or quietly reweights it, the thing a paying user made last month stops being reproducible this month, and I have no changelog to hand them.
Pinning a version only delays it. The pin gets deprecated too, usually with a few weeks of notice, and the replacement is better on the benchmark and different on the exact prompts people have built habits around. Better is not the same as the same.
The part I don't have is detection. You'd need a fixed set of prompts running on a schedule with a person looking at the outputs, which is an eval suite for taste, and I've never seen one that works. So in practice we find out when someone writes in to say it used to look right.
If you're building on models you don't own, do you check for drift at all, or does a user tell you first?
Replies
Softorino 💻📲
Most of what I work on converts files locally so nothing underneath it can move, and reading this I am glad of that in a way I was not before. The closest thing I have seen work is not an eval suite but a folder of about twenty real user files run on a schedule, where a person only looks when the output size or duration jumps. That catches breakage and misses taste, which is probably the honest ceiling.
@yelyzaveta_kibets The size and duration tripwire is the bit I can actually steal, because it's cheap and it doesn't pretend to judge anything. We run something like it on video jobs now, frame count and file size, and it caught two bad weeks nobody wrote in about. Where it falls apart for me is image, where drift shows up as everything getting slightly more saturated and every file is exactly the size it was yesterday. Honest ceiling is the right phrase, I just wish the ceiling were higher