Bot identity, IDs, and categories
Arcjet identifies bots by ID, such as GOOGLE_CRAWLER, and groups them into
categories, such as CATEGORY:SEARCH_ENGINE. A detectBot rule allows or
denies bots by ID, by category, or both, and for each detected bot Arcjet Cloud
reports a bot identity that says what kind of client it is and how Arcjet
established it.
This page is the reference for bot IDs, categories, and the bot identity. For the scenarios, see Detect AI agents and Block malicious bots.
Pass exactly one of allow or deny on each rule. An empty allow list
blocks every detected bot. If you use TypeScript, bot IDs and categories appear
as autocomplete values for the allow and deny options. In Python, the same
string identifiers are accepted, and categories are also available through the
BotCategory enum. In Go, use the same strings or a BotCategory* constant
such as arcjet.BotCategorySearchEngine.
Individual bots
Section titled “Individual bots”The bot list contains a list of known bots that Arcjet can identify.
For example:
detectBot({ mode: "LIVE", // will block requests. Use "DRY_RUN" to log only // Block all bots except specific Google and Bing crawlers, and curl allow: [ "GOOGLE_CRAWLER", "GOOGLE_CRAWLER_NEWS", "BING_CRAWLER", "CURL", ],}),detect_bot( mode=Mode.LIVE, # Blocks requests. Use Mode.DRY_RUN to log only # Block all bots except specific Google and Bing crawlers, and curl allow=[ "GOOGLE_CRAWLER", "GOOGLE_CRAWLER_NEWS", "BING_CRAWLER", "CURL", ],)arcjet.DetectBot(arcjet.BotOptions{ Mode: arcjet.ModeLive, // Blocks requests. Use ModeDryRun to log only. Allow: []string{ "GOOGLE_CRAWLER", "GOOGLE_CRAWLER_NEWS", "BING_CRAWLER", "CURL", },})Bot categories
Section titled “Bot categories”Arcjet uses the following categories:
CATEGORY:ACADEMIC: Scrape data for research purposesCATEGORY:ADVERTISING: Scrape data for advertising and marketing purposesCATEGORY:AI: Scrape data for AI and LLM purposesCATEGORY:AI_AGENT: Fetch a page on demand for a person using an AI product. Every bot in this category is also inCATEGORY:AICATEGORY:AMAZON: Scrape data for Amazon products and servicesCATEGORY:APPLE: Scrape data for Apple products and servicesCATEGORY:ARCHIVE: Scrape data for archival purposesCATEGORY:BOTNET: Automated behavior for malicious botnetsCATEGORY:FEEDFETCHER: Request data for RSS and other feedsCATEGORY:GOOGLE: Scrape data for Google products and servicesCATEGORY:META: Scrape data for Meta/Facebook products and servicesCATEGORY:MICROSOFT: Scrape data for Microsoft products and servicesCATEGORY:MONITOR: Interact for monitoring purposesCATEGORY:OPTIMIZER: Interact for optimization purposesCATEGORY:PREVIEW: Request data for image and URL previewsCATEGORY:PROGRAMMATIC: Interact through programming language librariesCATEGORY:SEARCH_ENGINE: Index data for search enginesCATEGORY:SLACK: Scrape data for Slack products and servicesCATEGORY:SOCIAL: Scrape data for social media products and servicesCATEGORY:TOOL: Interact through command line and GUI toolsCATEGORY:UNKNOWN: Known bots that cannot be classifiedCATEGORY:VERCEL: Scrape data for Vercel products and servicesCATEGORY:WEBHOOK: Automated HTTP notifications for events, such as Stripe paymentsCATEGORY:YAHOO: Scrape data for Yahoo products and services
The bot list contains a list of categories and which bots are in each category.
detectBot({ mode: "LIVE", // will block requests. Use "DRY_RUN" to log only // Block all bots except search engines and curl allow: [ "CATEGORY:SEARCH_ENGINE", // Google, Bing, etc "CURL", // You can allow specific bots in addition to categories ],}),from arcjet import BotCategory, detect_bot, Mode
detect_bot( mode=Mode.LIVE, # Blocks requests. Use Mode.DRY_RUN to log only # Block all bots except search engines and curl allow=[ BotCategory.SEARCH_ENGINE, # Google, Bing, etc "CURL", # You can allow specific bots in addition to categories ],)The Python SDK also accepts the underlying string identifiers (for example,
"CATEGORY:SEARCH_ENGINE") if you prefer not to import the enum.
Allow AI agents and block AI crawlers
Section titled “Allow AI agents and block AI crawlers”Every bot in CATEGORY:AI_AGENT is also in CATEGORY:AI, so a rule that
denies CATEGORY:AI also blocks AI agents. To block AI crawlers and let AI
agents through, use an allow list that includes CATEGORY:AI_AGENT, plus the
other categories that your site needs. For examples in each language, see
Allow AI agents and block AI crawlers.
Coding agents such as Claude Code fetch pages from the developer’s own computer, not from the AI company’s servers. Arcjet can’t verify them against the company’s published IP addresses, so Arcjet reports these bots as neither verified nor spoofed.
Only configured categories are checked for performance reasons. Each detected
bot must be compared to a category, so the worst case performance is
count(detectedBot) * count(configuredCategories).
We’re continuously evaluating bots to decide whether any need reclassifying. If we determine enough bots exist for a new category, we’ll consider adding new ones. If you need a specific category, open an issue on the arcjet/well-known-bots repository.
Custom bots
Section titled “Custom bots”If a bot that is not on our list signs its requests with Web Bot Auth, you can
allow or deny it by its key directory URL, for example
SIGNED_BOT:https://bot.example.com/.well-known/http-message-signatures-directory.
For details, see Verify signed bots with Web Bot Auth.
To block a particular bot that is not on our list and doesn’t sign its requests, you can use Arcjet Filters. For how to block custom bots, see the Malicious traffic blueprint.
Detection
Section titled “Detection”Arcjet provides additional bot verification using IP analysis, which can help if you are under attack from bots pretending to be good bots, such as clients pretending to be Google (whom you usually want to allow).
Known bots structure
Section titled “Known bots structure”The identifiers on the bot list are generated from a collection of known bots which includes details of their owner and any variations.
We welcome contributions to the arcjet/well-known-bots repository, whether you’re adding new bots or updating detection patterns. After a change is merged, Arcjet updates its pinned copy of the catalog. The update reaches Arcjet Cloud when Arcjet deploys it, and the SDK’s local detection in the next SDK release. The SDK runs local bot detection in the Arcjet WebAssembly module bundled with the SDK, so new patterns must be compiled into the module as part of the release process.
For instructions on how to contribute to our bot detection, read the repository’s README.md.
Unknown bots
Section titled “Unknown bots”If a bot is detected but cannot be identified as a known bot, Arcjet labels it
UNKNOWN_BOT. This is separate from the CATEGORY:UNKNOWN category, which is
for bots that cannot be classified into any category but can still be identified
as a specific bot. You can see a list of these named, but unclassified bots in
the bot
list.
Detections returned as UNKNOWN_BOT happen if the bot is new or hides itself.
It’s a bot with no name. Arcjet uses various techniques to detect these bots,
including analyzing request patterns and tracking IP addresses.
If you configure an allow rule and do not include UNKNOWN_BOT, then Arcjet
blocks detected bots that it cannot identify. This is the default behavior to
protect against new and rapidly evolving bots.
If you want to be more permissive and only block known, named bots, you can
include UNKNOWN_BOT in your allow rules.
Bot identity
Section titled “Bot identity”When Arcjet Cloud detects a bot, the bot rule’s reason includes one primary bot identity: who Arcjet thinks sent the request, and how it knows. A request can match several bots, so the identity is the detected bot with the strongest evidence. If two bots have equally strong evidence, Arcjet prefers an AI agent, then an AI crawler. The JavaScript, Python, and Go SDKs expose the identity.
Only Arcjet Cloud computes the identity. It’s unset when no bot was detected, and decisions that the SDK makes locally, such as a denial from the SDK’s local bot detection, never have one.
The identity has the following fields. In Python, botId is bot_id. In Go,
each field name starts with a capital letter, such as BotID.
| Field | Description |
|---|---|
kind | What kind of client the bot is: AI_AGENT, AI_CRAWLER, AUTOMATION, or OTHER. |
botId | The bot ID, such as CHATGPT_AGENT or a SIGNED_BOT: directory. Empty for AUTOMATION. |
operator | The hostname that operates the bot, such as chatgpt.com. Set only when the evidence is SIGNATURE. |
delegated | true when the agent presented a verified Personal Agent Protocol session, so it acts for a user who delegated access to it. Set only when the evidence is SIGNATURE. The user is never identified. |
evidence | How Arcjet established the identity: UNVERIFIED, FINGERPRINT, IP, or SIGNATURE. |
spoofed | true when IP address or reverse DNS verification showed that the request didn’t come from the bot’s operator. Arcjet checks this only when it has data for the IP address. |
To read the identity, check each rule result: result.reason.identity in
JavaScript, result.reason_v2.identity in Python, or
result.Reason.Bot.Identity in Go.
The kind field has the following values:
AI_AGENT: An AI product that fetches a page on demand for a person who is waiting on the answer. The bot is inCATEGORY:AI_AGENT.AI_CRAWLER: An AI bot that collects pages ahead of time, for model training or an AI search index. The bot is inCATEGORY:AIbut not inCATEGORY:AI_AGENT.AUTOMATION: Browser automation that advanced signals detected, with no named bot.OTHER: Any other detected bot.
The evidence field has the following values, from weakest to strongest:
UNVERIFIED: Arcjet matched the bot by itsUser-Agentheader or by IP reputation, and didn’t verify it. A spoofed identity always has this evidence.FINGERPRINT: The advanced signals browser checks failed.IP: The IP address is in the operator’s published ranges, Arcjet’s IP data names the bot as a known crawler, or reverse DNS confirmed it.SIGNATURE: A Web Bot Auth signature verified against the operator’s key directory, or a Personal Agent Protocol session verified against the agent’s published keys.
A bot identity tells you who sent the request, not whether to allow it.
UNVERIFIED is only a User-Agent claim, and SIGNATURE identifies the
operator of the signing key directory, not the end user. For details, see
A bot identity isn’t authorization.
Check for an AI agent
Section titled “Check for an AI agent”To check whether a request came from an AI agent, use isAiAgent from
@arcjet/inspect in JavaScript, is_ai_agent in Python, or
decision.IsAIAgent() in Go. The helper returns true when a LIVE bot rule
reports an identity with kind AI_AGENT that isn’t spoofed. It ignores
DRY_RUN results and doesn’t check the evidence, so read evidence when the
strength of the identity matters to you. For complete examples, see
Customize the response for AI agents.