Same bug, same prompt, three AI coding tools: two wrote the identical fix

by•

A small test worth copying when choosing between coding tools: give each one the same real bug and the exact same prompt, each in its own clean copy of the project, and score them against a checklist written before the first run.

The bug: a Next.js app for YouTube creators rejected some normal YouTube links with "Could not extract a video ID". Claude Code and GitHub Copilot both ran Claude Sonnet 5.5. Cursor ran Grok 4.6, so its result isn't a tool-only comparison.

What happened:

- Claude Code and Copilot produced byte-for-byte the same fix. Same model, same prompt: the tool wrapped around it changed very little.

- A hidden check of 8 link shapes, never mentioned in the prompt, gave all three the same 6 of 8. All three missed embed links and live links.

Three takeaways:

  1. On a small fix, the model matters more than the app it runs in.

  2. It works on the link in the bug report" says little. List the input shapes before prompting, then test them all.

  3. A prompt that names only the failing case tends to get a fix for that case only.

One bug and one run each, so it's a data point, not a ranking.

How would you word the prompt so embed and live links get covered without listing every format?

Full run with the diffs and the scorecard:

Disclosure: from my own channel

6 views

Add a comment

Replies

Be the first to reply

Have a question or a thought to share? Add a comment above to start the conversation.