Who actually verifies the LLM generated QA tickets?
Our PO runs an llm over the codebase and the dev environment and it files QA issues straight into jira. they show up in my queue already written up.
i still read every single one by hand.
the Wording is t he first thing. not wrong, but just written for another model rather than a person, so i end up rewriting the ticket in my head before i can even judge it.
Bigger issue is nothing checks them before they land. plenty aren't frontend at all, they're backend or model behavior, but they got filed to me anyway. and some are just not true. asks for a fix on something that isn't broken, or wants me to undo something we did on purpose.
not blaming the PO here, the tool did what it was asked to do. but the verifying landed on me and that step didn't exist before.
handing the actual fix to an LLM once i've confirmed it, that part genuinley works. its the confirming that doubled the job.
Has anyone solved the middle bit? not a prompt thing. i'm wondering whether people put a human review before it becomes a ticket, or just gave up and eat the cost.
Replies
I think having a human do the final QA is still important because they can catch issues AI might mis before users do
@shivam_kushwaha16 yeah fair. the human part isnt really what i'm complaining about, it's that it sits at the very end for us.moving it one step earlier is probably the whole fix.
thanks, this helped me name the actual problem 😃
@seenew Yeah that makes sense Moving the human review one step earlier could catch those issues before they turn into bigger problems Glad it helped
If the PO runs the scan, the PO turns each finding into a repro before it's a ticket: steps, expected, actual. Anything that can't be reproduced in the dev environment never reaches Jira (or Rally, etc).
When a decision is deliberate, I pin it with a test whose name says why. An agent that wants to "fix" it hits a red test with the reason in it, and a ticket asking for the same thing gets the same answer.
Backend issues landing in your queue is a triage problem, and I wouldn't let a model do triage for a team until someone had checked of its guesses
@siarheihamanovich um this is really useful thanks!
the test with the reason in the name is going straight in. we already have e2e so i can pin the deliberate ones there instead of re-explaining them every sprint.
going to pitch the repro gate too. framing it as triage rather than a QA problem is probably how i get that conversation to actually land 🤔