What an agent does when you tell it which testing technique to use
Dan Luu ran an experiment we wish we had run. One task, a Zstd implementation in Rust from the spec, given to a coding agent 160 times per condition, where a condition is, in most cases, one line appended to the prompt: use test-driven development, use QuickCheck, use fuzzing, use TLA+, audit first. Then he scored each run on a hidden test suite. That was the whole design.
No instruction wildly beat the run with none. He had written down six guesses beforehand about which conditions would not outperform, and all six came true. What interested us was how the instructed runs failed, because the failures share a shape. Told to use QuickCheck, all of them used the library, and 63 of 160 checked a single property, mostly with random inputs that hit the same rejection path. Told to do differential testing, 135 of 160 did something resembling it, and none built a second full implementation; where it might have caught something, the agent wrote the same code twice and put the same bug in both. Told to use TLA+, 159 of 160 wrote a model, and he could not find one case where the model changed the code. Told to fuzz, 10 of 160 built structured inputs, and about half of those found a real bug, which he counts among the more effective uses of any technique he saw.
Our reading is that every verification technique earns its value from one place: a check that did not come from the implementation being tested, such as a second implementation the agent did not write, or a property that existed before the code did. Naming the technique gives the agent the motions of the technique. The check still comes from the agent's own reading of the spec, so the new framework fills up with the tests it would have written anyway, and the suite it generates tells you what the agent already believed about the code. Our own data point is smaller: on a function with a sixteen-year-old bug, the agent-written suites that saw the broken code all passed, and two of them wrote the bug in as the expected result.
The rule we took away is to ask, before naming a technique, where its independent check will come from in this task, and then to supply that, or set up the structure that makes the agent go and get it.
Is RAG quietly dying? What we're actually shipping instead in 2026
Two years ago RAG was the default answer to "how do I make an LLM know my data." Chunk it, embed it, retrieve top-k, stuff it in context. Every tutorial, every framework, every "AI wrapper" startup pitch deck had the same diagram.
I don't think that default holds anymore, and I've been rebuilding around it for the past few months.
Three things broke the old assumption for me:
Context windows got cheap. When you can dump 200k+ tokens for pennies, "retrieve only the relevant chunk" stops being the obvious optimization - it becomes a lossy compression step you're paying latency and accuracy for, not saving cost on.
Chunking is still the weakest link nobody wants to admit. Semantic chunking, sentence-window, parent-child - every trick papers over the same problem: you're guessing where meaning boundaries are before you know the query. Agentic/iterative retrieval (let the model decide what to fetch next, over multiple turns) beats static top-k retrieval on every eval I've run, just at higher latency cost.
Fine-tuning got more accessible again. For narrow, stable domains, a lightly fine-tuned small model + light retrieval now often beats a heavy RAG pipeline on a frontier model - cheaper per query and more consistent.
How do you balance marketing and self-advocacy against your personal brand and reputation?
I think a lot of the PH community can relate that we have a lot of pride in our projects/software and want as many people as possible to see and use it. As a result, we often self-promote across platforms like Reddit, LinkedIn, X, etc.
On the one hand, I call this self-advocacy, because no one else will post about you and what you've dedicated hundreds of hours to. On the other hand, self-promoting too much can lead to reputational harm, getting marked as a spammer, and being lost in the sea of other "self-promoters."
How do you find this balance? I think I tend towards being too conservative. Any tips on successfully getting your message/product out there without being a shill?
Final thought: it feels like self-promotion is okay for already successful companies, but up and comers are seen as spammers. Bit of a catch 22?
for People who use scrapers a lot, what are the most common pain points?
recently i was checking out the scraper services and it seemed like easy to build at first but when i thought about scaling it, that's when it got really difficult.
still i wanted to know the opinions of the people who use the scraper services and what services do they use commonly in general and their pain point, that way i will just focus on those points and try to build the first iterations.
really appreciate your thoughts on this matter.
It’s time to start designing for agents, not humans
For as long as software has existed, the user has been a person. Someone sitting at a desk, poking at a phone, or calling an API. That assumption was so obvious it was never really a design principle, it was just common sense. Every decision about hierarchy, color, button placement, and error messaging was downstream of a single fact: a human being is going to see this.
With agents, that assumption is starting to fail.
Would you let AI send your emails without seeing them first?
Genuine question, not rhetorical.
Every AI writing tool is drifting toward autonomy ... read the thread, draft the reply, send it, tell you afterwards. The demos are genuinely impressive.
Should AI agents earn more authority over time
While designing approval flows for AI agents, I ve started questioning static permissions.
A better model might be:
New agent: approval required
Proven on low-risk actions: more autonomy
High-impact or irreversible actions: always human-reviewed
Performance drops: authority gets reduced again
Basically, treat autonomy as something an agent earns through reliable performance, not something we configure once.
Over- Designing basic flows is ruining Web Tools....
Had to download an invoice from a SaaS tool I use.
Simple enough right? Except the billing section was hidden inside a profile settings dropdown buried under an "Account Health" tab.
The interface looks incredibly polished. Modern fonts, pristine layout, clean icons. But finding a single PDF took way longer than it should have.
We're accidentally creating a generation of salespeople who can't sell
I've been watching this weird trend in B2B sales, and it's honestly keeping me up at night.
Everyone's rushing to implement AI sales tools (rightfully so - the efficiency gains are insane), but we might be accidentally breaking the entire profession.
Should AI products let users control how much freedom the AI has?
While working on an AI product recently, I ran into a product-design question that turned out to be much more important than I expected: how much freedom should the AI have when the user gives it constraints?
I initially assumed that better AI meant giving the model more freedom to find the best answer. But in practical workflows, that isn t always what users want, and sometimes the constraint itself is the point.