Skip to content

Bot identity, IDs, and categories

Arcjet identifies bots by ID, such as GOOGLE_CRAWLER, and groups them into categories, such as CATEGORY:SEARCH_ENGINE. A detectBot rule allows or denies bots by ID, by category, or both, and for each detected bot Arcjet Cloud reports a bot identity that says what kind of client it is and how Arcjet established it.

This page is the reference for bot IDs, categories, and the bot identity. For the scenarios, see Detect AI agents and Block malicious bots.

Pass exactly one of allow or deny on each rule. An empty allow list blocks every detected bot. If you use TypeScript, bot IDs and categories appear as autocomplete values for the allow and deny options. In Python, the same string identifiers are accepted, and categories are also available through the BotCategory enum. In Go, use the same strings or a BotCategory* constant such as arcjet.BotCategorySearchEngine.

The bot list contains a list of known bots that Arcjet can identify.

For example:

detectBot({
mode: "LIVE", // will block requests. Use "DRY_RUN" to log only
// Block all bots except specific Google and Bing crawlers, and curl
allow: [
"GOOGLE_CRAWLER",
"GOOGLE_CRAWLER_NEWS",
"BING_CRAWLER",
"CURL",
],
}),

Arcjet uses the following categories:

  • CATEGORY:ACADEMIC: Scrape data for research purposes
  • CATEGORY:ADVERTISING: Scrape data for advertising and marketing purposes
  • CATEGORY:AI: Scrape data for AI and LLM purposes
  • CATEGORY:AI_AGENT: Fetch a page on demand for a person using an AI product. Every bot in this category is also in CATEGORY:AI
  • CATEGORY:AMAZON: Scrape data for Amazon products and services
  • CATEGORY:APPLE: Scrape data for Apple products and services
  • CATEGORY:ARCHIVE: Scrape data for archival purposes
  • CATEGORY:BOTNET: Automated behavior for malicious botnets
  • CATEGORY:FEEDFETCHER: Request data for RSS and other feeds
  • CATEGORY:GOOGLE: Scrape data for Google products and services
  • CATEGORY:META: Scrape data for Meta/Facebook products and services
  • CATEGORY:MICROSOFT: Scrape data for Microsoft products and services
  • CATEGORY:MONITOR: Interact for monitoring purposes
  • CATEGORY:OPTIMIZER: Interact for optimization purposes
  • CATEGORY:PREVIEW: Request data for image and URL previews
  • CATEGORY:PROGRAMMATIC: Interact through programming language libraries
  • CATEGORY:SEARCH_ENGINE: Index data for search engines
  • CATEGORY:SLACK: Scrape data for Slack products and services
  • CATEGORY:SOCIAL: Scrape data for social media products and services
  • CATEGORY:TOOL: Interact through command line and GUI tools
  • CATEGORY:UNKNOWN: Known bots that cannot be classified
  • CATEGORY:VERCEL: Scrape data for Vercel products and services
  • CATEGORY:WEBHOOK: Automated HTTP notifications for events, such as Stripe payments
  • CATEGORY:YAHOO: Scrape data for Yahoo products and services

The bot list contains a list of categories and which bots are in each category.

detectBot({
mode: "LIVE", // will block requests. Use "DRY_RUN" to log only
// Block all bots except search engines and curl
allow: [
"CATEGORY:SEARCH_ENGINE", // Google, Bing, etc
"CURL", // You can allow specific bots in addition to categories
],
}),

Every bot in CATEGORY:AI_AGENT is also in CATEGORY:AI, so a rule that denies CATEGORY:AI also blocks AI agents. To block AI crawlers and let AI agents through, use an allow list that includes CATEGORY:AI_AGENT, plus the other categories that your site needs. For examples in each language, see Allow AI agents and block AI crawlers.

Coding agents such as Claude Code fetch pages from the developer’s own computer, not from the AI company’s servers. Arcjet can’t verify them against the company’s published IP addresses, so Arcjet reports these bots as neither verified nor spoofed.

Only configured categories are checked for performance reasons. Each detected bot must be compared to a category, so the worst case performance is count(detectedBot) * count(configuredCategories).

We’re continuously evaluating bots to decide whether any need reclassifying. If we determine enough bots exist for a new category, we’ll consider adding new ones. If you need a specific category, open an issue on the arcjet/well-known-bots repository.

If a bot that is not on our list signs its requests with Web Bot Auth, you can allow or deny it by its key directory URL, for example SIGNED_BOT:https://bot.example.com/.well-known/http-message-signatures-directory. For details, see Verify signed bots with Web Bot Auth.

To block a particular bot that is not on our list and doesn’t sign its requests, you can use Arcjet Filters. For how to block custom bots, see the Malicious traffic blueprint.

Arcjet provides additional bot verification using IP analysis, which can help if you are under attack from bots pretending to be good bots, such as clients pretending to be Google (whom you usually want to allow).

The identifiers on the bot list are generated from a collection of known bots which includes details of their owner and any variations.

We welcome contributions to the arcjet/well-known-bots repository, whether you’re adding new bots or updating detection patterns. After a change is merged, Arcjet updates its pinned copy of the catalog. The update reaches Arcjet Cloud when Arcjet deploys it, and the SDK’s local detection in the next SDK release. The SDK runs local bot detection in the Arcjet WebAssembly module bundled with the SDK, so new patterns must be compiled into the module as part of the release process.

For instructions on how to contribute to our bot detection, read the repository’s README.md.

If a bot is detected but cannot be identified as a known bot, Arcjet labels it UNKNOWN_BOT. This is separate from the CATEGORY:UNKNOWN category, which is for bots that cannot be classified into any category but can still be identified as a specific bot. You can see a list of these named, but unclassified bots in the bot list.

Detections returned as UNKNOWN_BOT happen if the bot is new or hides itself. It’s a bot with no name. Arcjet uses various techniques to detect these bots, including analyzing request patterns and tracking IP addresses.

If you configure an allow rule and do not include UNKNOWN_BOT, then Arcjet blocks detected bots that it cannot identify. This is the default behavior to protect against new and rapidly evolving bots.

If you want to be more permissive and only block known, named bots, you can include UNKNOWN_BOT in your allow rules.

When Arcjet Cloud detects a bot, the bot rule’s reason includes one primary bot identity: who Arcjet thinks sent the request, and how it knows. A request can match several bots, so the identity is the detected bot with the strongest evidence. If two bots have equally strong evidence, Arcjet prefers an AI agent, then an AI crawler. The JavaScript, Python, and Go SDKs expose the identity.

Only Arcjet Cloud computes the identity. It’s unset when no bot was detected, and decisions that the SDK makes locally, such as a denial from the SDK’s local bot detection, never have one.

The identity has the following fields. In Python, botId is bot_id. In Go, each field name starts with a capital letter, such as BotID.

FieldDescription
kindWhat kind of client the bot is: AI_AGENT, AI_CRAWLER, AUTOMATION, or OTHER.
botIdThe bot ID, such as CHATGPT_AGENT or a SIGNED_BOT: directory. Empty for AUTOMATION.
operatorThe hostname that operates the bot, such as chatgpt.com. Set only when the evidence is SIGNATURE.
delegatedtrue when the agent presented a verified Personal Agent Protocol session, so it acts for a user who delegated access to it. Set only when the evidence is SIGNATURE. The user is never identified.
evidenceHow Arcjet established the identity: UNVERIFIED, FINGERPRINT, IP, or SIGNATURE.
spoofedtrue when IP address or reverse DNS verification showed that the request didn’t come from the bot’s operator. Arcjet checks this only when it has data for the IP address.

To read the identity, check each rule result: result.reason.identity in JavaScript, result.reason_v2.identity in Python, or result.Reason.Bot.Identity in Go.

The kind field has the following values:

  • AI_AGENT: An AI product that fetches a page on demand for a person who is waiting on the answer. The bot is in CATEGORY:AI_AGENT.
  • AI_CRAWLER: An AI bot that collects pages ahead of time, for model training or an AI search index. The bot is in CATEGORY:AI but not in CATEGORY:AI_AGENT.
  • AUTOMATION: Browser automation that advanced signals detected, with no named bot.
  • OTHER: Any other detected bot.

The evidence field has the following values, from weakest to strongest:

  • UNVERIFIED: Arcjet matched the bot by its User-Agent header or by IP reputation, and didn’t verify it. A spoofed identity always has this evidence.
  • FINGERPRINT: The advanced signals browser checks failed.
  • IP: The IP address is in the operator’s published ranges, Arcjet’s IP data names the bot as a known crawler, or reverse DNS confirmed it.
  • SIGNATURE: A Web Bot Auth signature verified against the operator’s key directory, or a Personal Agent Protocol session verified against the agent’s published keys.

A bot identity tells you who sent the request, not whether to allow it. UNVERIFIED is only a User-Agent claim, and SIGNATURE identifies the operator of the signing key directory, not the end user. For details, see A bot identity isn’t authorization.

To check whether a request came from an AI agent, use isAiAgent from @arcjet/inspect in JavaScript, is_ai_agent in Python, or decision.IsAIAgent() in Go. The helper returns true when a LIVE bot rule reports an identity with kind AI_AGENT that isn’t spoofed. It ignores DRY_RUN results and doesn’t check the evidence, so read evidence when the strength of the identity matters to you. For complete examples, see Customize the response for AI agents.

Discussion