Where I put Jev in production on a large site, and where I didn't

I run a large job marketplace in Japan, and Jev has been in production on it for about 10 days. Not as a reviewer, it's making real calls in live jobs.

The first place was the company list we keep in house. We check each company name on it against the operator name on their website. Code settles most of them, and only the leftovers go to Jev, with just the two names (about 200 tokens). On the first run it settled 571 of 996 companies, and 50 hand checks found 0 mistakes. Since today it also reads replies in our inbox: is it a person, and does it need an answer.

Where I didn't put it: skipping the bigger model for support mail. It saved about 30%, but missed 3 messages we actually wanted to answer. Also anything code can just read off the page, like a copyright line. Asking about that only makes a sure thing unsure.

What I learned is to ask one small fact per question. Bundle two claims with "or" and the scores pile up around 0.75, give it a whole page and 93% land in the middle. Send only the part that decides it. And ask twice, since about 1 in 6 answers flipped between runs.

One caution. For Jev to decide, you have to send it the content. We sent inbox and support mail, but not member messages. Their servers are in the US, so that would need to be in our privacy policy first.

Where have you drawn the line?