What would you let your coding agent A/B test first?
Hi, I’m Andrei, the maker of SplitHub. We’re launching our coding-agent workflow on October 15.
A coding agent can build three versions of a page quickly. Choosing which version actually helps still needs real traffic and a well-defined conversion goal.
SplitHub gives Claude Code, Cursor and Codex skills, CLI and MCP tools to create experiments, manage them, and read the results. React and Next.js SDKs assign variants locally. The SDK is MIT licensed; self-hosting provides raw counts, while SplitHub Cloud adds statistical analysis and verdicts.
Explore the dashboard without signing in: https://app.splithub.io/demo
The shop and data are fictional; the results are processed by the real statistics engine. Try checkout-button for a clear winner, free-shipping-banner for a refund guardrail warning, and pricing-cta for a test without a convincing winner.
What would you test first — a pricing headline, an onboarding step, a checkout button, or something else? And which decisions would you want to approve yourself before your agent changes an experiment?
Replies
I'd start with the checkout or signup button copy, since it's low-risk, easy to measure, and traffic is usually enough to reach a result in a reasonable time. Pricing headlines and onboarding steps are more interesting but need more care. For what I'd approve myself: anything that changes the primary goal metric or the guardrail metrics (refunds, churn, support tickets), stopping or "shipping" a winner, and any change to who gets bucketed into the test. I'd be happy to let the agent create variants and draft the setup, but declaring a winner is the step where a human should sanity-check that the sample size and goal were sensible. One practical thing worth having is a pre-set minimum runtime, so an agent can't call a test early just because the first two days looked good.