Why I now test AI prompts with examples
I used to judge an AI prompt by reading its first response. If the result looked good I assumed the prompt was ready to use.
That approach missed a problem. The same prompt could produce different results with different inputs. A prompt that worked for a support question might fail when a customer described three problems at once.
Now I collect, around 10 examples before changing a prompt. I include cases, unusual requests, incomplete information and examples that previously produced poor answers.
For each example I write down what a good result should contain. A support reply might need the policy, a clear next step and a calm tone without promising something the company cannot provide.
Then I compare the new prompts using the same examples. I check accuracy and consistency than choosing whichever response sounds more polished.
This small test set also becomes useful later. When I change the model or add instructions I can run the examples again. See whether the results improved or became worse.
My takeaway: Keep a collection of real test cases and reuse it after every important prompt change. How examples do you test before putting an AI workflow into production?
Replies
Same lesson here, plus one addition: run each example more than once. A model can pass a case once and fail it the next time with the same input. When a song in my app showed the wrong name in its lyrics, I first suspected the step that writes them, ran it eight times on that request, and it got the name right 8 of 8. The real culprit was the next step, the one that aligns the lyrics to the sung audio. After the prompt fix it kept the name 3 of 3. Writing results as counts instead of "looks good" is what told me which prompt to fix in the first place.