AI content safety

AI applications accept text from users and generate text of their own. Either side can contain harmful content: a user can ask for sexual or hateful material, and a model can produce it even from an innocent prompt.

Arcjet content moderation checks text for harmful content before you send it to a model, store it, or show it to a user. It’s a Guard rule, so you call it with guard() at the point where the text crosses a boundary in your application.

Why content safety matters

Harmful content in prompts and outputs is a product problem and a compliance problem:

Payment processors and card networks set rules for businesses that use their payment rails, and those rules cover generated content. An AI application that generates sexual or suggestive material can fall out of compliance and risk fines or losing the ability to take payments.
App stores review apps for objectionable content. An app that shows user-generated or AI-generated text needs a way to filter it.
Your users see what your model returns. Harmful output that reaches a customer damages trust in your product.

The default content moderation policy enforces sexual content most strictly because that’s the main risk for payment processor compliance. It also covers misconduct, hate speech, and insults. For more information about the categories, see the content moderation policy.

Moderate input and output

Check the user’s message before it reaches the model, then check the model’s reply before you show it. Configure the rule once and reuse it for both checks. Use a different label for each check so you can tell input and output decisions apart in the Arcjet Console.

JavaScript / TypeScript
Python

1
import { launchArcjet, moderateContent } from "@arcjet/guard";
2

3
const arcjet = launchArcjet({ key: process.env.ARCJET_KEY! });
4
const moderate = moderateContent();
5

6
export async function chat(userId: string, message: string) {
7
  // Check the user's message before it reaches the model
8
  const inputDecision = await arcjet.guard({
9
    label: "chat.input",
10
    metadata: { userId },
11
    rules: [moderate(message)],
12
  });
13

14
  if (inputDecision.conclusion === "DENY") {
15
    return "This message can't be processed. Rephrase it and try again.";
16
  }
17

18
  // Replace with your model call
19
  const reply = await generateReply(message);
20

21
  // Check the model's reply before you show it
22
  const outputDecision = await arcjet.guard({
23
    label: "chat.output",
24
    metadata: { userId },
25
    rules: [moderate(reply)],
26
  });
27

28
  if (outputDecision.conclusion === "DENY") {
29
    return "This response can't be shown. Try a different request.";
30
  }
31

32
  return reply;
33
}

import os

from arcjet.guard import ModerateContent, launch_arcjet

arcjet = launch_arcjet(key=os.environ["ARCJET_KEY"])
moderate = ModerateContent()


async def chat(user_id: str, message: str) -> str:
    # Check the user's message before it reaches the model
    input_decision = await arcjet.guard(
        label="chat.input",
        metadata={"user_id": user_id},
        rules=[moderate(message)],
    )

    if input_decision.conclusion == "DENY":
        return "This message can't be processed. Rephrase it and try again."

    # Replace with your model call
    reply = await generate_reply(message)

    # Check the model's reply before you show it
    output_decision = await arcjet.guard(
        label="chat.output",
        metadata={"user_id": user_id},
        rules=[moderate(reply)],
    )

    if output_decision.conclusion == "DENY":
        return "This response can't be shown. Try a different request."

    return reply

Keep denied responses generic. Don’t explain exactly what was flagged, and don’t copy the checked text into metadata.

Combine with prompt injection detection

Content moderation checks whether text is harmful. Prompt injection detection checks whether text tries to override your model’s instructions. A user message can do either, so check both in the same guard() call:

JavaScript / TypeScript
Python

1
import {
2
  detectPromptInjection,
3
  launchArcjet,
4
  moderateContent,
5
} from "@arcjet/guard";
6

7
const arcjet = launchArcjet({ key: process.env.ARCJET_KEY! });
8
const moderate = moderateContent();
9
const detectInjection = detectPromptInjection();
10

11
export async function checkMessage(userId: string, message: string) {
12
  const decision = await arcjet.guard({
13
    label: "chat.input",
14
    metadata: { userId },
15
    rules: [moderate(message), detectInjection(message)],
16
  });
17

18
  // Returns true when the message can go to the model
19
  return decision.conclusion !== "DENY";
20
}

import os

from arcjet.guard import DetectPromptInjection, ModerateContent, launch_arcjet

arcjet = launch_arcjet(key=os.environ["ARCJET_KEY"])
moderate = ModerateContent()
detect_injection = DetectPromptInjection()


async def check_message(user_id: str, message: str) -> bool:
    decision = await arcjet.guard(
        label="chat.input",
        metadata={"user_id": user_id},
        rules=[moderate(message), detect_injection(message)],
    )

    # Returns True when the message can go to the model
    return decision.conclusion != "DENY"

The decision is DENY when either rule detects a problem. For more information about prompt injection, see AI prompt injection detection.

Handle errors

If Arcjet can’t complete a content moderation check, the rule returns an error result and the call is allowed, so a service issue doesn’t block your application. If your application can’t accept unchecked text, check decision.hasFailedOpen() in JavaScript or decision.has_failed_open() in Python and deny the action yourself. For more information, see Errors and timeouts.

AI content safety

Why content safety matters

Moderate input and output

Combine with prompt injection detection

Handle errors

Learn more