How do you decide when an AI draft is good enough to send to a customer?
This question has sat in the back of my head for months and I still don't have a clean answer.
For internal work the bar is forgiving. A draft brief with rough edges, a slightly off tone in a Slack summary, a CRM note that's 80% right. You fix it and move on.
The second that output goes to a customer, something shifts. I read it three times looking for anything that could land wrong. Often the output is fine. The issue isn't quality, it's that I don't fully trust the process yet, and I'm not sure that's irrational.
My rule for now: reversible and internal can run on its own. Anything customer-facing or relationship-affecting, I see before it goes. The cost of a mistake is different depending on where it lands.
What I haven't worked out is where "low-stakes customer-facing" sits. An automated confirmation email feels fine. A cold follow-up feels different. A proposal feels completely different again.
Where do you draw the line, and has it moved the longer you've used AI at work?
Replies
My situation is a strange one, because my product itself is AI output that goes to the customer with no human in between: someone describes a song, a model writes and sings it, and they hear it a minute later. Nobody reviews it, by design, and people are fine with that because they asked for it and they know what made it. The line I draw is about words that are supposed to come from me. A support reply is drafted by a model from a file of facts about the product, in the person's language, but it never goes out until I've read their message and edited the draft. The draft is usually fine. What it can't know is what I'd promise and what I wouldn't, and that is the only part of a customer message that actually matters. So my rule is close to yours, just phrased differently: if the reader believes a human wrote it, a human has to have read it. A confirmation email passes because nobody thinks a person typed it. A cold follow up fails because it pretends to be personal. Has the line moved? Yes, in one direction: the more drafts I read, the fewer I change, but I haven't stopped reading, and I don't think I will.
i think the line is less customer facing vs internal and more can the mistake be walked back. a wrong confirmation email is cheap, a proposal with a wrong promise is not. seems like the line moves when the draft shows where each claim came from. do you reread the whole thing or check it against the source?
Our version of "low-stakes customer-facing" is the support chat on our SEO platform. An AI assistant answers only if nobody on the team has replied within 60 seconds. Every reply it sends is copied to me, and the moment I answer myself it goes quiet for 30 minutes. If someone asks "are you a bot?", it says yes.
What moved the line for us wasn't accuracy, it was memory. Early on it reopened a complaint that had been settled weeks before, because it read a months-long chat as one conversation. Now it only answers what came after the last reply. So to Hiten's question: I check a draft against what was already said to that person, not just against the source.