I needed some fairly specific content moderation tooling for another project I'm working on, capable of capturing things like fallacies, cognitive distortions, persuasiveness, factual correctness, on-topic-ness, etc on top of more traditional moderation categories (i.e. harmful content). A lot of the existing services have fairly narrow categories for the content they can screen for, and tend to feel more enterprise / unapproachable (as a surface-level example, most want you to call them for a price). So I decided to build my own, starting w/ a pool of small, open LLMs as the basis for the consensus system, and threading them through more traditional ML approaches to draw conclusions from their individual votes. I was very happy with the results, and found that it worked w/ fairly high accuracy out-of-the-box for any category. With the success of the system I decided to roll it out as an independent service. Many of the categories within its intended use-case are judgement calls where reasonable people could disagree, so I added a customization component, allowing the system to recognize the kinds of questions where its answers might disagree with the user's preferences, and adjust accordingly. Personalization is factored in at request time, so when a user submits feedback on a verdict or provides examples, the effect will be felt from the very next evaluation.

CriteriaBot

The Universal, Customizable Classifier

The Universal, Customizable Classifier