Framer AI AgentsDesign and publish professional sites with AI
Qlane
Merge PRs with confidence - AI QA that runs your whole app
41 followers
Merge PRs with confidence - AI QA that runs your whole app
41 followers
Most tools sit on one side of a line. AI code-review tools read the diff but never run it, so they guess "this could be null." Black-box E2E tools drive a real browser but see only the DOM. Qlane boots your whole app in a sandbox and tests it from the inside - browser, shell, source, plus connectable logs and database - so it reports only bugs that actually reproduce, and traces each to its root cause. runs your app, not just your code.
"Reading code and running code catch completely different bugs" is the thesis I'd tattoo on a wall — I verify my own changes by driving the actual app for exactly this reason, because a green typecheck and a working app are different claims. The question that decides how much I'd trust it: fidelity of the sandbox. Booting the stack with Docker Compose and seeded data catches "does the happy path run," but the bugs only running catches usually live in production-shaped state — an auth-token refresh race, a connection pooler that refuses you under a specific condition, a third-party API returning something your seeds never do. Seeded data almost by definition doesn't contain the state that produced the bug. Can Qlane snapshot real (anonymized) state into the ephemeral env, or is it always synthetic seeds? That gap is the difference between "the PR runs" and "the PR is safe."
@narek_keshishyan Really sharp question, and honestly, this is the edge of what we do today.
The sandbox is whatever your Compose + seed step define. The seed step is an arbitrary command you own, so you can already restore an anonymized dump instead of synthetic fixtures and get production-shaped state. First-class "snapshot prod state into the ephemeral env" isn't wired yet - there's literally a slot for it in our config schema waiting for the implementation. It's the right ask.
What surprised us in practice: a large share of running-only bugs don't need exotic state - cross-service contract breaks, migrations, API/DB mismatches reproduce on any data, but only when the app actually runs. The token-refresh-race class you describe is real and harder.
Curious: would an anonymized snapshot even clear compliance on your side, or would you need synthetic-but-production-shaped data?
the "boots your whole app and actually runs it" part is the compelling piece, but that cuts both ways - it means the agent can trigger real side effects too, not just observe them. if a PR touches checkout, does the run risk hitting a payment provider's live mode or sending a real email/webhook, or is that sandboxed away by design? feels like the same power that catches real bugs could also cause real damage if it's not scoped carefully.
@galdayan Fair concern! The gate is credentials: the sandbox receives only the env vars you configure for that environment (encrypted at rest, decrypted into that one run), so a checkout PR talks to whatever your config points at - Stripe test mode, a mock, staging. It can't reach live mode unless you literally hand it live keys. The sandbox is ephemeral and torn down when the run ends.
The agent side is scoped too: the browser stays on the target app, and its shell is an allowlisted read-only toolkit - it investigates, it doesn't mutate.
For some integrations (like Stripe) we are going to have skills and tools, which you can enable and qlane will use Stripe Test Clock to test end-to-end lifecycle of payments and subscriptions. Skills engine is already in place an working - we have pre-built ones for accessibility, auth, etc. You can even use your custom skills there.
makes sense, thanks for walking through it. the "read-only" shell part is what I'm curious about next - if a test needs the app itself to write to its own DB or filesystem (which e2e almost always does), is that happening inside the sandbox unrestricted, and "read-only" only applies to the agent's own investigation tools poking around outside the app's normal write paths?
Exactly right - two different planes.
The app itself runs unrestricted inside its own containers: when the agent clicks "checkout", the app writes to its DB, filesystem, queues - whatever it normally does. That's the point: the state changes are real, which is also why the bugs it finds come with real server-log lines behind them.
"Read-only" is the agent's own toolkit. Its shell is an allowlisted investigation kit - read source, tail logs, probe endpoints - with file mutation, package installs, and write-shaped commands structurally blocked, not just prompted away. So the agent can only change state the way a user would: through the front door of your app.
And since the sandbox is torn down after the run, everything the app wrote dies with it - no cleanup scripts, no polluted staging DB.
@leestex That's a clean separation, and the ephemeral teardown answers my main worry about state pollution. One more edge case: since the app runs unrestricted inside its own sandbox, what happens to any real third-party side effects it triggers - a payment provider, an email service, a webhook to some external system? Is that on the dev to point their env config at sandboxed/test keys before running Qlane, or does it do anything to intercept outbound calls before they leave the container?
How does this handle apps with heavy external service dependencies, like third-party APIs that need real credentials during a test run?
@satkcaix I love the question! This literally went from theory to production for us last week.
The base layer: env vars are configured per environment, encrypted at rest, decrypted only into that run's sandbox. So third-party APIs get whatever credentials you give them, ideally test-mode keys or a mock. The app just sees a normal .env.
But some dependencies need more than keys. Auth providers validate your app's public URL, and our sandboxes get ephemeral URLs on a domain that sits on the Public Suffix List - so WorkOS (as an example) refuses wildcard redirect URIs for it, and real login inside a test run was impossible to allowlist. We just shipped custom domains for sandboxes: a stateless edge proxy fronts the sandbox on a custom (non-PSL) domain, set per environment; you register https://*.your-domain/callback once in the vendor's dashboard, and the whole OAuth round-trip (real sign-in form included) happens inside the sandbox.
And we're cooking something on exactly this front that should make testing against external integrations quite a bit more flexible. What's the heaviest external dependency in your stack?
As a solo dev shipping AI-written code daily, this hits a real pain: unit tests pass but the actual app breaks. Question — can it catch visual regressions? In my product (image processing) the worst bugs are "output looks subtly wrong but nothing throws," and I've only been able to cover those with hand-written pixel-level regression tests.
@kojimajunya Honest answer: partly today, more soon. And I'd keep your pixel tests either way.
Today the agent drives the real running app and catches what surfaces in behavior - wrong output, error states, broken flows, console and network errors - with a screenshot attached to every bug as evidence. It doesn't pixel-diff against a baseline yet.
What's coming next is visual comparison built for PRs: the agent captures the same flows on the PR's base and on its head, then diffs the screenshots to flag unintended visual changes. Because both sides render in the same sandbox minutes apart, same fonts, same rendering stack - the cross-environment flake that plagues screenshot testing mostly disappears, and there's no baseline gallery to maintain or rot.
Your case is extra interesting because the output IS an image. The same mechanism applies: run the same fixture images through both sides and compare outputs with a perceptual tolerance instead of exact pixels. That catches "this PR subtly changed the output" without hand-maintained goldens - your golden tests would still own absolute correctness. Would base-vs-head on a fixture corpus cover most of your cases, or do you specifically need absolute goldens?
@leestex Base-vs-head on a fixture corpus would honestly cover ~80% of my cases — most of my regressions are exactly "this PR subtly changed the output." So yes, that mechanism would be valuable on day one.
But here's a real bug from my project that base-vs-head can't catch by design: I once shipped wrinkle shading with inverted polarity (highlights and shadows swapped) due to a misordered compositing argument. The PR that introduced it did change the output — but it was a PR that was supposed to change the output, so a visual diff just confirms "yes, shading changed," which tells you nothing about whether the new shading is physically right. The output looked plausible, so it passed human review too. And once it's merged, every later PR renders it identically wrong on both sides — zero diff, forever invisible. I only caught it with an assertion about the physics ("shadow regions must be darker than the mean"), not about pixels changing.
So my split would be: your base-vs-head diff owns "did this PR change anything unintentionally" (the high-frequency case), and a small set of property-based assertions owns absolute correctness (the rare-but-nasty case). If your agent could someday generate those property assertions from a natural-language spec ("shadows darken, highlights brighten, print edges stay sharp"), that'd replace my hand-written goldens entirely.
Looking forward to the PR visual diff — that alone would replace a chunk of my manual review.
how does it handle flaky stuff like network requests or third-party api calls when reproducing bugs in the sandbox?
@ouzhangdmodira Two layers. The environment is deterministic - a fresh sandbox per run, your compose stack at the PR's exact commit, so there's no shared-state drift between runs; third-party calls hit whatever credentials you configured, which keeps them stable too. And the reporting bar is observation, not inference: a bug has to be seen live in the browser or the server logs, with repro steps and a screenshot attached - so you can always judge whether it was a real failure or a blip.
How does it handle flaky tests or external services like third-party APIs when spinning up the whole sandbox?
@elmasnerz9d8 External services get exactly the credentials you configure per environment - test-mode keys or mocks, same as you'd set up staging. On flakiness: every run is a fresh sandbox booted from your own compose file at the PR's commit, and a bug only gets reported when the agent actually observed the failure - repro steps and screenshot attached, never inferred from the diff.