Content moderation (AI)
Also known as: moderation · safety moderation
Content moderation is detecting and filtering unsafe or policy-violating content — whether in what users send, what models output, or media they generate.
- Usually a combination: a classifier model + rule filters + sampling review.
- Applied on both sides: inputs (jailbreak attempts) and outputs (harmful content).
- A layer in defense in depth, not a single wall.
Expect false positives: moderation optimizes for catching bad content, which means some harmless requests get blocked. Understanding the layer helps you phrase around legitimate refusals — and know when a refusal is policy vs. an error.