Introduction to Arcjet content moderation

Arcjet content moderation evaluates untrusted text for harmful content before it is stored, displayed, or forwarded to another service. Use it on user messages, tool-call results, and model outputs at the same action boundary as Agent guards.

What is Arcjet?

Arcjet is the AI agent runtime security platform. Discover the agents running in your organization, enforce policy across every action, prompt, and tool call, and keep the evidence to prove what happened. Detect prompt injection, authorize agent tool calls, redact PII, and block bots and abuse.

Content moderation is a Guard rule. There is no protect() / HTTP-request variant. Call it from guard() in JavaScript or Python, or Guard in Go.

When to use content moderation

Use content moderation when you need a decision on text that is about to leave your control or be shown to a user, for example:

• User-facing chat and support assistants – block harmful messages before they are stored or displayed.
• Agent tool results and model outputs – scan generated text before it is forwarded to another tool, a customer, or a third-party API.
• Moderation at the action boundary – the same guard() call can combine content moderation with rate limiting or prompt injection detection.

Content moderation answers a different question from prompt injection detection (hostile instructions aimed at the model) and sensitive information detection (PII and other regulated data). Combine them when a single untrusted string needs more than one check.

Guard-only rule

Content moderation inspects caller-supplied text, so it is configured in code with the Guard SDK. It is not available as a remote rule or on protect().

How Arcjet content moderation works

1. You configure moderateContent() (JavaScript), ModerateContent() (Python), or GuardModerateContent (Go) once and reuse the rule.
2. At the action boundary, bind the untrusted text and submit it with guard() / Guard.
3. Arcjet returns ALLOW or DENY. A denial has reason MODERATE_CONTENT.
4. The per-rule result is a binary detected / Detected verdict plus optional billing in text_units.

The result shape does not include per-category scores. For what the default policy enforces and why it focuses on sexual content for payment processor compliance, see the content moderation policy.

SDK availability

SDK	Public API	Notes
JavaScript / TypeScript	moderateContent()	mode is optional and defaults to LIVE.
Python	ModerateContent()	mode is optional and defaults to LIVE.
Go	GuardModerateContent	Mode is required (ModeLive or ModeDryRun). An empty Mode returns ErrInvalidMode.

Examples

Configure the rule once, bind the untrusted text, and submit it with guard() / Guard:

JavaScript / TypeScript

import { launchArcjet, moderateContent } from "@arcjet/guard";

const arcjet = launchArcjet({ key: process.env.ARCJET_KEY! });
const moderate = moderateContent();

const decision = await arcjet.guard({
  label: "tools.chat",
  rules: [moderate(userMessage)],
});

if (decision.conclusion === "DENY" && decision.reason === "MODERATE_CONTENT") {
  throw new Error("Harmful content detected – rephrase your message");
}

const result = moderate.result(decision);
// `detected` is true when harmful content was found. Billing is undefined
// when the service does not report usage. Content moderation uses text_units.
console.log(result?.detected, result?.billing?.unit, result?.billing?.count);

Python

import os

from arcjet.guard import ModerateContent, launch_arcjet

arcjet = launch_arcjet(key=os.environ["ARCJET_KEY"])
moderate = ModerateContent()

decision = await arcjet.guard(
    label="llm.output",
    rules=[moderate(text)],
)

if decision.conclusion == "DENY" and decision.reason == "MODERATE_CONTENT":
    raise RuntimeError("Harmful content detected – rephrase your message")

result = moderate.result(decision)
# `detected` is True when harmful content was found. Billing is None
# when the service does not report usage. Content moderation uses text_units.
print(result.detected if result else None)
if result and result.billing:
    print(result.billing.unit, result.billing.count)

Go

moderation, err := arcjet.GuardModerateContent(arcjet.GuardModerateContentOptions{
  Mode: arcjet.ModeLive, // required
})
if err != nil {
  return err
}

decision, err := guard.Guard(ctx, arcjet.GuardRequest{
  Label: "tools.generate",
  Rules: []arcjet.GuardRuleInput{moderation.Text(userMessage)},
})
if err != nil {
  return err
}
if decision.IsDenied() && decision.Reason == arcjet.ReasonModerateContent {
  return errors.New("content flagged by moderation")
}

// Billing is optional. Content moderation usage is measured in text_units.
if result := moderation.Result(decision); result != nil && result.Billing != nil {
  fmt.Printf("charged %d %s\n", result.Billing.Count, result.Billing.Unit)
}

Set Mode on every Guard rule. An empty Mode returns ErrInvalidMode.

For install steps and framework-specific wrappers, see the content moderation quick start.

Dry run mode

mode: "DRY_RUN" / ModeDryRun evaluates the rule without blocking. Use this to measure detections before switching to "LIVE".

Errors and timeouts

If Arcjet can’t complete the check in time, or the text to check is empty, the rule returns an error result instead of a verdict. An error result doesn’t deny, so the call is allowed. This keeps a service issue from blocking your application.

To detect an error result, use decision.hasFailedOpen() in JavaScript, decision.has_failed_open() in Python, or decision.HasFailedOpen() in Go. Log the error, and deny the action yourself if your application can’t accept unchecked text. For the error codes, see Prompt injection and content moderation rule codes.

Data handling

Content moderation runs on Arcjet, so the text that you pass to the rule is sent to the Arcjet Cloud API for evaluation. Arcjet doesn’t store the text. The decision keeps the verdict, the billing count, and the label and metadata that you attach to the call, so don’t copy the text into metadata. For what else Arcjet stores and for how long, see Arcjet privacy.

Pricing

Content moderation is priced based on usage, measured in text_units when the service reports billing. A text unit is the length of the text you submit to a single moderation call, in characters, divided by 1,000 and rounded up. For a worked example, see how text units are calculated. For the price per text unit, see the pricing page.

Discussion