I started treating AI experiments like product experiments, not just prompts

I've noticed that when I use AI for a business problem, it's very easy to keep changing the prompt until the output looks good.

That creates a weird problem.

You can end up optimizing the answer without knowing whether you've improved the process.

So I've started treating AI experiments more like product experiments.

Before changing anything, I write down:

What am I trying to improve?

For example:

Then:

What does the current process produce?

And:

What would count as a meaningful improvement?

Only after that do I start changing the prompt, workflow, model, or automation.

It sounds obvious, but it changes the experiment.

Instead of:

Prompt → better-looking answer → "seems better"

it's:

Baseline → change one thing → compare → keep or revert

I've found this especially useful because AI outputs can feel better without actually being more useful.

A longer answer isn't necessarily better research.

A more confident answer isn't necessarily more accurate.

A faster workflow isn't necessarily better if it creates more checking later.

I'm curious how other builders evaluate their AI experiments:

Do you keep a baseline and compare results, or mostly judge the output by whether it feels better than the previous attempt?

29 views