Anyone actually checked their vibe-coded app for security holes, or just hoping for the best?
Been thinking about this after seeing a few posts here about vibe-coded apps breaking in production. Auth bugs, webhook failures, that kind of thing.
What worries me more is the stuff that doesn't break loudly. No error, no crash, it just quietly leaks data or accepts requests it shouldn't. Things like exposed API keys sitting in the frontend bundle, missing rate limits, or database rules that let any logged in user read rows that aren't theirs. None of that throws an error. It just sits there until someone finds it.
For the non-engineers building real products with this stuff, what does your actual security check look like before you launch. Are you running anything to scan for this, paying someone to review it once, asking the AI itself to audit its own code, or just shipping and hoping nothing bad happens.
Curious if there is an actual workflow people are settled on, or if this is still the part everyone quietly skips.
Replies
The scary bugs aren't the ones that crash your app they're the ones that quietly work exactly as an attacker hopes. 😅
@craig_bennett1 Can't agree more. A crash is annoying but at least it's honest, it tells on itself immediately. The stuff that just works quietly in someone else's favor can sit there for months with nobody noticing, and by the time it does show up it's not a bug report, it's a breach.
Toggling RLS on isn't the same as testing it, the boundary only gets exercised once you log in as a second, different user and try to touch the first user's row from that session. And 'denied' has two flavors worth telling apart: an empty result, the policy filtering correctly, versus a permission error, a missing grant. Same outcome at a glance, different meaning.
@vollos That distinction matters more than it looks. A permission error usually means a missing grant, not the policy actually filtering rows, and if someone fixes the grant later without checking the policy logic, that's when the real leak shows up. How do you usually tell the two apart, error code, or testing with a role that has the grant but should still get filtered?
@vollos @elhart05 Testing with a role that has the grant but should still get filtered is the only one of the two that actually tells you anything. Error code alone can't distinguish the two failure modes, because it's not actually testing the same thing.
A missing grant fails before the query even runs — Postgres throws permission denied for table X (SQLSTATE 42501) at the parse/authorization stage. RLS filtering doesn't error at all. The query succeeds, returns 200, and just silently returns zero rows or a subset. Those look completely different in an error log, but if you're testing with a role that lacks the grant, you never reach the policy logic in the first place — you're testing Postgres's coarse table-level ACL, not the fine-grained predicate you actually care about.
So the only way to verify the policy itself is doing its job is: grant the role table access, then check whether the filter still applies. If a "should be blocked" query returns 200 with an empty array when it should return 200 with an empty array because of the policy — not because the role has no access — that's the actual thing you're trying to prove. Concretely, SET ROLE to the test role and run EXPLAIN (ANALYZE, VERBOSE) on the query; if RLS is active you'll see the filter condition applied in the plan even before rows are counted. pg_policies is worth checking too, just to confirm the predicate you think is attached is actually the one that's live.
The failure mode you're describing — someone fixes a grant later without re-checking policy logic — is exactly the case a grant-based test would never have caught, because the grant-based test was never exercising the policy to begin with. It's the same shape of mistake as trusting "is this secure" instead of checking the artifact: a clean permission-denied error feels like proof, but it's proof of a different, less important thing than the one that actually matters.
@elhart05 @actuallydone Testing with a role that has the grant but should still get filtered, Christian's right, that one proves something real. I leaned on error codes for a while and got burned: had a WITH CHECK typo that let writes through, but every permission-denied log during testing looked clean because the test role didn't even have the grant yet. Only caught it once I gave the role real access and logged in as the second user.
@elhart05 @vollos That's a sharp catch, and a genuinely dangerous failure mode — a policy that looks airtight because your test conditions never actually exercised it.
I hit a version of this on Actually Done's per-user data isolation. Early on I was testing access control by hitting endpoints as an unauthenticated or wrong-role user and confirming I got denied. All green. But that's the same trap you're describing — denial-by-default isn't proof the policy logic is correct, it's just proof the door was locked before I tried the key. It wasn't until I deliberately logged in as a second real user with actual legitimate access and tried to reach the first user's data that a cross-user leak in one endpoint actually surfaced.
Now it's a hard rule for any permission or isolation work: test with a principal that has the grant, not just ones that don't. "Access denied" in a log tells you the guard fired — it says nothing about whether the guard's logic is right.
Real answer: I didn't have a formal security workflow starting out, and I think that's the honest state for most non-engineers doing this. But a few things happened along the way that became my actual practice, not because I planned it, but because I got burned into it.
The big one: I had Claude and the Agent audit the codebase directly, and we found a real cross-user data leak — a housekeeping endpoint that let one user's request touch another user's rows. No error, no crash, exactly the "sits there until someone finds it" problem you're describing. We also found a case where a null field silently dropped transaction records due to a LEFT JOIN + WHERE interaction — again, no error, just wrong behavior.
What actually caught these wasn't asking the AI "is this secure?" in the abstract — it was asking it to audit specific categories: auth boundaries, data isolation between users, and anywhere user input touches a query. General "review your code" prompts are too vague and miss this stuff. Specific categories work better.
For anything sensitive, I went further than trusting the AI's word — I pulled up the production database directly via psql and looked at the raw ciphertext myself to confirm encryption was actually happening, not just claimed. That's probably the single most useful habit: don't accept "it's secure" as an answer, verify it yourself at the data layer when you can.
I don't think there's a settled industry workflow yet for non-engineers. What's worked for me: targeted audits by category (not vague "check my code"), direct verification of anything security-critical instead of trusting the report, and treating "no error" as meaningless — plenty of the worst bugs never throw one.
@actuallydone The category specific audits point is the real insight here, vague 'is this secure' prompts just get a reassuring paragraph back. Checking the raw ciphertext yourself instead of trusting the AI's word on encryption is probably the best habit in here too. Did this turn into a repeatable checklist for you, or still more per project?
@elhart05 It became a checklist, pretty explicitly. What started as one-off skepticism on the Password Vault (raw DB audit) turned into standing rules that apply to every feature now: no "fixed" claims without screenshot proof, zero open issues before republish, root cause not patches, batch fixes into one message. Those aren't per-project judgment calls anymore — they're just how every Agent report gets handled by default.
The category-specific instinct is newer but heading the same way. "Is this secure" would've gotten a paragraph. Asking for the raw ciphertext, or later asking the Agent to show the actual DNS record / webhook event count / screenshot of a real received email — that's the same move each time: don't accept a description of the fix, make it show the artifact. Tonight's Resend thing is a good example — the Agent's own audit turned up a real gap (welcome email) sitting next to a non-issue dressed up like a gap (Stripe receipts). The checklist habit is what caught that distinction instead of just reacting to "two items still not sending."
So yeah — less "how do I verify this feature" and more "what's the category of claim, and what's the artifact that actually proves it." That transfers cleanly across encryption, DNS, webhooks, whatever's next.