Skip to content

Detect AI agents and block AI crawlers

An AI agent is an AI product that fetches a page or takes an action for a person who is waiting on the result, such as ChatGPT fetching a page that a user asked about. Arcjet detects AI agents with the detectBot rule, tells them apart from AI crawlers that collect pages for model training, and reports how it identified each one.

With that information, you can let AI agents use your site, keep AI crawlers out, and decide what each agent gets. The examples on this page use @arcjet/node, FastAPI, and net/http. The same rules and helpers work in Next.js and the other JavaScript SDKs.

AI traffic falls into two groups, and most sites want to treat them differently:

KindWhat it doesCategoryExamples
AI agentFetches a page or acts on demand, for a person who is waiting on the answer.CATEGORY:AI_AGENTOPENAI_CRAWLER_USER (ChatGPT-User), CHATGPT_AGENT, ANTHROPIC_CRAWLER_USER (Claude-User), PERPLEXITY_USER, GOOGLE_AGENT, ANTHROPIC_CLAUDE_CODE
AI crawlerCollects pages ahead of time, for model training or an AI search index.CATEGORY:AIOPENAI_CRAWLER (GPTBot), ANTHROPIC_CRAWLER (ClaudeBot), PERPLEXITY_CRAWLER, COMMONCRAWL_CRAWLER

AI agents reach your site in several ways:

  • Fetchers request a single page when a person asks a chatbot about it, such as ChatGPT-User, Claude-User, and Perplexity-User.
  • Cloud browsers run a full browser on the AI company’s servers to complete a task, such as ChatGPT agent (CHATGPT_AGENT) and Google’s agent (GOOGLE_AGENT).
  • Hosted browser platforms run browser sessions for other companies’ agents, such as Browserbase (BROWSERBASE) and Manus (MANUS_BOT).
  • Coding agents fetch documentation from a developer’s own computer, such as Claude Code (ANTHROPIC_CLAUDE_CODE), Cursor (CURSOR), and GitHub Copilot (GITHUB_COPILOT).

Every bot in CATEGORY:AI_AGENT is also in CATEGORY:AI. For the full list of bot IDs, see Bot identity, IDs, and categories.

To keep crawlers out and let agents through, use an allow list that includes CATEGORY:AI_AGENT. A rule that denies CATEGORY:AI also blocks AI agents, because every agent is in that category too.

An allow list blocks every other detected bot, so also include the categories that your site relies on, such as CATEGORY:SEARCH_ENGINE. Some AI search crawlers, such as OAI-SearchBot and YouBot, are also in CATEGORY:SEARCH_ENGINE, so allowing that category lets them through.

detectBot({
mode: "LIVE", // Blocks requests. Use "DRY_RUN" to log only
// Block all bots except search engines and AI agents
allow: [
"CATEGORY:SEARCH_ENGINE", // Google, Bing, etc
"CATEGORY:AI_AGENT", // ChatGPT-User, Claude-User, coding agents, etc
],
}),

To block AI agents as well as crawlers, add CATEGORY:AI to a deny list. To block one agent, add its bot ID to a deny list.

Any client can claim to be ChatGPT-User in its User-Agent header, so Arcjet checks the claim where the operator gives it a way to:

  • Published IP address ranges. OpenAI, Anthropic, Perplexity, and Google publish the addresses that their fetchers use. A ChatGPT-User request from one of OpenAI’s addresses has IP evidence.
  • Web Bot Auth signatures. ChatGPT agent, Google’s agent, DuckAssistBot, Browserbase, Manus, and other operators sign each request with a key that they publish on their own domain. A verified signature has SIGNATURE evidence and names the operator, such as chatgpt.com. For the full list, see Verify signed bots with Web Bot Auth.
  • Personal Agent Protocol sessions. A personal agent that acts for a signed-in or anonymous user sends a signed session with each request. A verified session has SIGNATURE evidence, names the agent’s operator, and sets delegated. For more information, see Agents that act for a user.

When a request claims to be an AI agent but its IP address isn’t in the operator’s ranges, Arcjet marks the identity as spoofed. A spoofed identity always has UNVERIFIED evidence, and the AI agent helpers in the following section return false for it.

Coding agents such as Claude Code fetch pages from the developer’s own computer, not from the AI company’s servers. Arcjet can’t verify them against published addresses, so it reports them with UNVERIFIED evidence, as neither verified nor spoofed.

Blocking or allowing is only the first choice. Once you know that a request came from an AI agent, you can give it a response that suits an agent and protects your site, such as the following:

  • Serve a lighter version of the page, such as Markdown without scripts and layout, which costs you less to serve and is easier for the agent to read.
  • Apply a separate rate limit to each agent, so that one agent can’t use up capacity that your other visitors need.
  • Return a helpful message that points the agent to your API instead of a page that it would have to scrape.
  • Require a person to sign in for account actions, such as purchases, password changes, and data deletion.

To check for an AI agent, use isAiAgent from @arcjet/inspect in JavaScript, is_ai_agent in Python, or decision.IsAIAgent() in Go. The helper returns true when a LIVE bot rule reports an identity with kind AI_AGENT that isn’t spoofed. It ignores DRY_RUN results and doesn’t check the evidence, so read the identity’s evidence field when the strength of the identity matters to you.

The JavaScript, Python, and Go SDKs expose the bot identity and the AI agent helpers. Only Arcjet Cloud reports the identity, so a decision that the SDK makes locally, such as a denial from its local bot detection, never has one.

The following example allows search engines and AI agents. It gives each AI agent its own rate limit, keyed by the bot ID, keeps account actions for signed-in people, and serves agents a Markdown version of the page:

index.ts
import arcjet, {
detectBot,
slidingWindow,
type ArcjetBotIdentity,
type ArcjetDecision,
} from "@arcjet/node";
import { isAiAgent } from "@arcjet/inspect";
import http from "node:http";
const aj = arcjet({
key: process.env.ARCJET_KEY!,
rules: [
detectBot({
mode: "LIVE", // Blocks requests. Use "DRY_RUN" to log only
// Block all bots except search engines and AI agents
allow: ["CATEGORY:SEARCH_ENGINE", "CATEGORY:AI_AGENT"],
}),
],
});
// A separate budget for each AI agent, keyed by its bot ID
const agentLimit = arcjet({
key: process.env.ARCJET_KEY!,
characteristics: ["botId"],
rules: [slidingWindow({ mode: "LIVE", interval: "1m", max: 60 })],
});
// Return the identity of the AI agent that sent the request, if any
function aiAgent(decision: ArcjetDecision): ArcjetBotIdentity | undefined {
for (const result of decision.results) {
if (isAiAgent(result) && result.reason.isBot()) {
return result.reason.identity;
}
}
return undefined;
}
const server = http.createServer(async (req, res) => {
const decision = await aj.protect(req);
if (decision.isDenied()) {
res.writeHead(403, { "Content-Type": "application/json" });
res.end(JSON.stringify({ error: "Forbidden" }));
return;
}
const agent = aiAgent(decision);
if (!agent) {
res.writeHead(200, { "Content-Type": "text/html" });
res.end("<h1>Pricing</h1><p>The full page for people.</p>");
return;
}
// For example: CHATGPT_AGENT SIGNATURE chatgpt.com
console.log(agent.botId, agent.evidence, agent.operator);
const limit = await agentLimit.protect(req, { botId: agent.botId });
if (limit.isDenied()) {
res.writeHead(429, { "Content-Type": "application/json" });
res.end(JSON.stringify({ error: "Too many requests" }));
return;
}
// Keep account actions for signed-in people and your API
if (req.url?.startsWith("/account")) {
res.writeHead(401, { "Content-Type": "application/json" });
res.end(
JSON.stringify({
error: "Sign in to manage an account.",
api: "https://api.example.com/docs",
}),
);
return;
}
// Serve a lighter Markdown version of the page
res.writeHead(200, { "Content-Type": "text/markdown" });
res.end("# Pricing\n\nThe same content, without scripts or layout.");
});
server.listen(3000);

The rate limit uses the bot ID as its key, so all requests from one agent share one budget, whichever person they’re for. To limit each agent per IP address instead, add ip.src to the characteristics. For more information, see Rate limit configuration.

The Personal Agent Protocol lets a personal agent, such as Meta’s Muse, act on your site for a person who delegated access to it. Instead of signing in with the person’s password, the agent sends a Poppy-Session header with each request: a token signed by a key that the agent publishes on its own domain, and bound to the request’s method and URL.

Arcjet verifies the session for agents in the well-known bots catalog and reports the following identity:

  • evidence is SIGNATURE and operator is the agent’s domain.
  • kind is AI_AGENT and botId is the agent’s bot ID.
  • delegated is true.

A verified session proves which agent sent the request and that it acts for one of its users. It never identifies the user: the session carries an ID that is different for each site and contains no personal information. It also doesn’t prove that the person approved this action, so keep your own checks for purchases and other account changes.

An agent sends a session only to sites that publish a /.well-known/poppy.json file, which your authorization server provides along with the OAuth endpoints that the agent signs in through. Arcjet checks the session against the request; your authorization server checks the access token and its scopes. For more information, see the Personal Agent Protocol specification.

To give delegated agents more than other agents, check delegated on the identity. The following example extends the aiAgent helper in the preceding section:

const agent = aiAgent(decision);
if (agent?.delegated) {
// A verified Personal Agent Protocol session: the agent's bot ID,
// SIGNATURE, the agent's domain, and true
console.log(agent.botId, agent.evidence, agent.operator, agent.delegated);
}

delegated is only true with SIGNATURE evidence. A session that fails verification, or comes from an agent that isn’t in the catalog, doesn’t change the identity.

Some agents don’t send a User-Agent header that names them. A cloud browser or a hosted browser platform runs a real browser and sends a standard browser User-Agent header, and an agent that controls a browser on a person’s own computer looks like that person’s browser.

The following signals help with these agents:

  • Web Bot Auth signatures. If the operator signs its requests, Arcjet names the operator even though the User-Agent header looks like a browser.
  • Advanced signals. The browser checks can detect automation in the browser. Arcjet then reports an identity with kind AUTOMATION and FINGERPRINT evidence. This tells you that a program drives the browser, but not which agent it is. For more information, see Advanced bot signals.

If an agent sends no signature and passes the browser checks, Arcjet treats the request as a browser. Rate limits and your own authorization checks still apply.

Add a detectBot rule that allows or denies CATEGORY:AI_AGENT, then check each rule result with isAiAgent in JavaScript, is_ai_agent in Python, or decision.IsAIAgent() in Go. Read the identity’s botId and evidence fields to see which agent it is and how Arcjet identified it.

ChatGPT agent signs its requests with Web Bot Auth, and Arcjet reports it as CHATGPT_AGENT with SIGNATURE evidence and the operator chatgpt.com. ChatGPT’s on-demand fetcher, ChatGPT-User, has the bot ID OPENAI_CRAWLER_USER, and Arcjet verifies it against OpenAI’s published IP address ranges. Both are in CATEGORY:AI_AGENT.

What’s the difference between an AI agent and an AI crawler?

Section titled “What’s the difference between an AI agent and an AI crawler?”

An AI agent fetches a page for a person who is waiting on the answer, and an AI crawler collects pages ahead of time for model training or an AI search index. In Arcjet, agents are in CATEGORY:AI_AGENT, and crawlers are the rest of CATEGORY:AI.

Usually not on public pages, because an AI agent often acts for a real person who might become your customer. Allow agents where they help your visitors, rate limit them, and require sign-in for account actions. Block them on routes where automated access does harm, such as signup forms.

How do I block AI crawlers but allow AI agents?

Section titled “How do I block AI crawlers but allow AI agents?”

Use an allow list that includes CATEGORY:AI_AGENT and leaves out CATEGORY:AI. Also include the other categories that your site needs, such as CATEGORY:SEARCH_ENGINE, because an allow list blocks every other detected bot.

Yes, any client can send an AI agent’s User-Agent header. Arcjet checks the claim against the operator’s published IP addresses and Web Bot Auth signatures where they exist, marks a failed check as spoofed, and reports the evidence level, so you can trust a verified agent more than a claim.

Why does Arcjet report Claude Code as unverified?

Section titled “Why does Arcjet report Claude Code as unverified?”

Claude Code and other coding agents fetch pages from the developer’s computer, so their requests never come from the AI company’s published IP addresses. Arcjet reports them with UNVERIFIED evidence and doesn’t mark them as spoofed.

The Personal Agent Protocol is an open protocol for personal agents that act for a user. The agent identifies itself with a signed session on each request, and signs in through your OAuth server with scopes the user approves, instead of using the user’s password. Arcjet verifies the session and reports the agent with SIGNATURE evidence and delegated set to true.

Arcjet verifies sessions only from agents in the well-known bots catalog, which records each agent’s client_id. To add an agent, open a pull request with the client_id that its operator publishes.

Muse is Meta’s personal agent. It browses from a cloud browser and doesn’t name itself in the User-Agent header. Muse signs requests with Web Bot Auth from muse.ai, which Arcjet verifies and reports as META_MUSE with SIGNATURE evidence. If your site publishes a /.well-known/poppy.json file, Muse can also present a Personal Agent Protocol session, which Arcjet reports with delegated set to true. A Muse request with neither a signature nor a session looks like a browser, so use advanced signals and rate limits.

Instinct is a personal agent that browses from a cloud browser and doesn’t name itself in the User-Agent header or sign its requests, so Arcjet can’t name it from a request alone. Advanced signals detect browser automation, and rate limits and your own authorization checks still apply. Instinct is a design partner of the Personal Agent Protocol; when it sends a verified session, Arcjet reports it like any other delegated agent.

Manus signs the requests from its cloud browser with Web Bot Auth, and Arcjet reports them as MANUS_BOT with SIGNATURE evidence. The Manus Browser Operator extension runs in a person’s own browser, with their sign-in and IP address, so those requests look like that person’s browser.

Claude’s on-demand fetcher, Claude-User, has the bot ID ANTHROPIC_CRAWLER_USER, and Arcjet verifies it against Anthropic’s published IP address ranges. Claude Code has the bot ID ANTHROPIC_CLAUDE_CODE. Claude in Chrome runs in a person’s own browser, so its requests look like that person’s browser.

Perplexity’s on-demand fetcher, Perplexity-User, has the bot ID PERPLEXITY_USER, and Arcjet verifies it against Perplexity’s published IP address ranges. Comet is a browser that runs on a person’s own computer and sends a standard browser User-Agent header, so Arcjet treats it as a browser unless advanced signals detect automation.

Google’s agent sends Google-Agent in its User-Agent header and signs its requests with Web Bot Auth. Arcjet reports it as GOOGLE_AGENT, with SIGNATURE evidence when the signature verifies and IP evidence from Google’s published address ranges.

xAI’s agent sends grok-agent in its User-Agent header, and Arcjet reports it as XAI_GROK_AGENT with UNVERIFIED evidence, because xAI publishes no IP address ranges or signing keys for it.

Which AI agents look like a normal browser?

Section titled “Which AI agents look like a normal browser?”

Agents that run in a person’s own browser, such as Claude in Chrome, Perplexity Comet, and the Manus Browser Operator extension, send that browser’s requests. Agents that run a cloud browser without signing, such as Instinct, send a standard browser User-Agent header. For these agents, use advanced signals, rate limits, and your own authorization checks.

Discussion