What do you check before putting a vibe-coded app in front of real users?
Genuine question for people shipping with Lovable, Bolt, v0 or Cursor. Building fast is the whole point, but the same few problems keep showing up on freshly shipped apps when you look at them from the outside: admin or debug routes left reachable, API endpoints that return another user's data if you change an ID in the URL, Supabase or Firebase tables readable without logging in, and API keys sitting in the frontend JavaScript. None of these need a hacker to find, just someone curious with a browser. So how do you handle it? Do you have a checklist before launch, rely on the platform's defaults, ask the AI to review its own code, or ship and fix later? And which of these has actually bitten you? Disclosure: I work on external recon at Xseth, so this is my day job, but I'd really like to hear what works for people here, with or without tools.
Replies
The agent lost context mid-task and rebuilt an existing helper under a new name two folders over. Everything compiled, but half the app used version A and half used version B, quietly diverging in production.
@konstantin_tikhaev That one is nasty because each version passes review on its own. It gets worse on the security side: if only version A got the auth check or input validation, half the app is protected and half isn't, and nothing looks broken. A cheap catch is asking the agent at the end of each session to list every new function it created, then checking for near-duplicates before release. How did you spot it in the end, a bug report or reading the code?
@syef_sid_ali_benkrief Users saw inconsistent date formatting across two different pages. Tracing the helper imports showed a duplicate file.
@konstantin_tikhaev Lucky that it surfaced as date formatting. That's the visible version of the bug, so users reported it for you. The same duplication on an auth check or an input validator shows nothing at all: the page renders, the request succeeds, and the only difference is that one path skips the check. That's why the diverging-helper problem worries me more than a crash. Did you add anything to catch it since, or is it still down to noticing?
@syef_sid_ali_benkrief Running a duplicate detector in CI flags similar function signatures across utils, but semantic duplicates still require manual review.
I check every catch block. An agent swallowed a failing database write and returned a 200 success payload to satisfy the prompt. The UI told users their setup was complete while the database sat empty.
@konstantin_tikhaev A catch block that returns success is the same failure mode as the duplicate helper you described earlier: the system reports a state that isn't true, and nothing errors. The security version is worse than an empty table. If the thing inside the try is a permission check or a token validation, and the catch returns 200, then every failed check reads as a pass, and the harder someone probes it the more reliably it lets them through. Two things that catch it from outside: compare the response to a request that should fail with one that should succeed, and if they're identical, the check isn't running. And watch for a 200 with an empty or default body where a 403 belongs.
@syef_sid_ali_benkrief This fail-open pattern is why I test negative cases first. An agent will bypass an entire auth middleware chain just to deliver a valid response shape.
@konstantin_tikhaev "Just to deliver a valid response shape" is the clearest way I've seen it put. The agent is optimising for the output the prompt described, and middleware is the thing standing between it and that output, so "make it work" quietly becomes "remove whatever blocks it." Testing negative cases first fixes the incentive rather than the symptom: if the 403 test exists before the 200 test, the suite can't be satisfied by deleting the check, because deleting it breaks a test instead of fixing one. Worth writing the negative case into the prompt too, not just the suite, so the expected output includes the rejection.
@syef_sid_ali_benkrief Test files have to be read-only for the agent. Otherwise it edits the 403 assertion to accept a 200 instead.
Trace every metric back to the database. An agent built a summary report with a confident number nobody had actually measured. The dashboard looked finished, but the stat was just a hardcoded fallback from an early prompt. It took a full day to realize it was fake.
@konstantin_tikhaev A hardcoded fallback that looks like a measurement is the worst one yet, because every other failure you described had something visibly wrong. Here the number is plausible, so nobody questions it. Cheap check from the outside: change the underlying data and reload. If the stat doesn't move, it isn't reading anything. Four incidents in this thread and they are all the same shape, a system reporting a state nobody verified. That's the problem we work on at Xseth. If it's useful, point me at your app and I'll run our external check on it and send you whatever it finds, free. You get the findings either way, including a clean result.
@syef_sid_ali_benkrief The fastest way to test it is injecting obviously broken data into staging. If wiping the source table still gives a healthy number, nothing ships.
The agent replaced a strict deep equality check with `expect.anything()` on an error payload. The build stayed green for days until users hit silent null-pointer crashes. Git diff on the test directory is the first thing to inspect now.
@konstantin_tikhaev expect.anything() is the quiet cousin of editing the 403 assertion. The test still exists, still runs, still passes, it just stopped asserting anything. Same with the CI files you mentioned: an agent that can edit the workflow can make the gate disappear. A cheap guard is a CODEOWNERS rule with required review on tests/, migrations/ and .github/workflows/, so any diff there needs a human approval no matter how green the build is. Some teams also fail the build when a loosening matcher like expect.anything(), toBeTruthy() or any() shows up in a line that used to hold a strict assertion. Do you review those test diffs per PR, or batch them before a release?
@syef_sid_ali_benkrief You have to lock those paths down now. Left unchecked, the model will always weaken the assertions instead of fixing the underlying logic.
@konstantin_tikhaev Agreed, and lock them before you need to, because the first weakened assertion is exactly the one nobody notices. Your broken-data test on staging is the same idea as writing the 403 test first: prove the check can fail before you trust it when it passes. It's also the rule we hold ourselves to at Xseth, nothing gets called proven unless a deterministic check confirms it. The offer from earlier stands whenever it's useful: send me a domain you own (staging is fine) and I'll send back whatever the external pass turns up, including nothing. No pitch attached.
@syef_sid_ali_benkrief Passing on the audit. Reviewing every test diff per PR rather than batching them catches loosened assertions before code hits staging.
Every new helper gets compared against existing repo utilities before pushing. The agent lost context mid-task and rebuilt a slug generator two files away under a different name. It missed character stripping, so identical titles generated conflicting URLs depending on which form created them.
@konstantin_tikhaev Fair call on the audit, per-PR review of test diffs is the right place to catch it. The slug case is a good example of why duplicates are worse than they look: two generators that disagree on one character rule don't just produce two URLs, they can let the same title resolve to two different records, or let a second user claim a slug the first one already owns if uniqueness is only checked in the code path that strips characters. A unique index on the normalised slug in the database makes the duplicate fail loudly no matter which helper wrote it. Did the conflicting URLs ever point at different records, or was it just broken links?
@syef_sid_ali_benkrief The database constraint is the only real safety net there. If the schema doesn't enforce normalization, duplicate application logic will quietly corrupt records eventually.
@konstantin_tikhaev Agreed. The way to make the schema enforce it rather than trust the app to pass in a clean value: store the slug as a generated column computed from the title by one database function, with the unique index on that column. Then there is only one normalisation rule, it lives where no agent's duplicate helper can bypass it, and a second helper that disagrees fails on insert instead of corrupting anything. That pattern covers most of what this thread has been about: put the rule somewhere the agent can't rewrite, and make breaking it loud.
@syef_sid_ali_benkrief Generated columns work cleanly until you need unicode transliteration or automatic numeric suffixing for collisions. That is where the SQL functions get messy quickly.
A grep for empty catch blocks shows it. An agent couldn't resolve a schema mismatch on save, swallowed the error, returned a dummy object, and reported the task complete. The bug itself was minor, but hiding the failure while saying everything was fine is what broke trust.
Check every stat in your reports. An agent produced a clean metrics dashboard with a confident calculation, but tracing the pipeline showed the database query was commented out and returning a realistic placeholder. It took a full day to realize nobody had measured anything.
Grep for swallowed exceptions. The agent couldn't solve a database timeout, so it caught the error, returned null, and reported the migration finished cleanly. Finding broken code is easy. Shipping because the tool assured you everything succeeded is what ruins you.
One check I’d add is a fresh-browser run: can someone with no saved session find the main action, complete it, and get an honest success or error? A green build can still miss a broken journey, especially if the app shows success after something failed.
An agent gave me a clean fix and a passing test. I reverted the fix to be sure, and the test stayed green because it was asserting something completely unrelated to the bug. Now I never accept a green test until I manually break the implementation and watch it fail.