We Benchmarked Whether a Pattern Reference Actually Changes How an AI Agent Refactors

Hi everyone,

We spend our days watching AI coding agents refactor legacy code, and we kept seeing the same failure. Agents know design patterns from training data, but they know them unevenly. They invent one-off abstractions where a named pattern would fit. They force patterns where they don't belong. And they skip the small, safe steps that keep tests green. So we built a pattern reference that an agent can pull mid-task. It's a website, an MCP server, and a set of agent skills. Then we asked ourselves a simple question. Does it actually change what the agent produces?

The Benchmark

We took one deliberately messy 840-line TypeScript codebase and copied it twice. Same agent, same model, same prompt. One copy had our reference, the other had nothing. We scored both results with the same static analyzer.

• The agent with the reference removed 100 lines of code. The one without it removed 53.

• Duplication dropped from 6.2% to 4.1% with the reference, and only to 4.9% without it.

• Every original test passed, and the typecheck was clean in both runs.

Honest small print: this was one run per condition on a small codebase, so treat it as a signal and not as proof. Our guess is the effect grows with codebase size, because on 840 lines the model already knows most of the answers. A bigger benchmark across more models is coming next.

What we built

The reference lives at https://aipatterns.dev. It has 158 pages of patterns, refactoring techniques, and code smells, plus 13 MCP tools and five agent skills your agent can use mid-task. It's free.

We'd love your thoughts

• Do named patterns and treatments actually improve your agents' refactors, or is it noise at your scale?

• What would make a test like this convincing? More repetitions, a bigger codebase, different metrics?

• Which code smell does your agent commit most? Ours was duplicated logic.