Has a “6x faster” benchmark ever turned out to be a bug in your own setup?
I recently spent a few weeks making a text-embedding service fast enough for a voice assistant. The budget was about 10 ms per sentence, and my first number was 150 to 170 ms.
My first instinct was to blame the language or the runtime. It wasn’t either:
Rust vs Python tied, because over 99.8% of the time was inside the model.
The real problem was the model file. The weights were stored as 8-bit but converted back to 32-bit on every single call. One offline script took it from about 150 ms to 34 ms, with identical output.
My own earlier report had credited OpenVINO with a 6.4x speed-up. Most of that was the same bug, so I went back and corrected it.
What I took from it: profile before you change anything, and be ready to correct your own benchmarks.
Curious about yours:
What’s the biggest win you ever got from a profiler?
Have you ever had to retract your own “X is N times faster” claim?
Which tool do you reach for first when something is slow?
https://mutedmiles.space/post/falling-into-the-embedding-city

Replies
Be the first to reply
Have a question or a thought to share? Add a comment above to start the conversation.