Do you track new AI model releases and test them as they come out?
by•
We're taking part in the GPT-6 Astra Challenge with OpenAI right now, which got me thinking about this.
New models drop constantly. Some people test everything immediately, others stick with what works until there's a real reason to switch.
Do you test new models right away? Do you use different ones for different tasks?
And if AI is part of your product - how do you decide when it's actually worth migrating?
50 views
Replies
Scarlett.
Absolutely. I test new models almost every time they drop.
Main reason: in my experience, older models often get noticeably worse once a new one launches. So testing the newest model is usually the most practical approach.
Fable was incredible when it first came out, then the quality dropped hard. I’ve noticed a similar pattern with some OpenAI models too.
So I don’t get too attached to any model. I keep testing and use whatever works best right now.
Glam AI
@byalexai Thanks for sharing, makes sense! Any specific model that surprised you recently?
AINA
For me, the model always depends on the task.
When Astra came out, we tested how personalized the coach’s final advice was after a session. The difference was huge.
Before, changing the model didn’t change the result much. We spent more time improving the prompt, but the results were usually similar.
With Astra, I read the final advice and thought: “This really feels like my personal action plan.” For e, this is when changing the model is worth it.
Glam AI
@kseniia_ponomareva Yeah, when it's part of your product, testing is probably obligatory. And things move so fast, you have to stay on your toes constantly :) Do you feel like it puts a lot of pressure on product/tech teams - having to work at that speed?