Identifying bots

Arcjet lets you configure a list of bots to allow or deny. To construct the list, you can specify individual bots, use categories to allow or deny all bots in a category, or both.

If you are using TypeScript, these appear as autocomplete values for the allow or deny options while writing your rules. In Python, the same string identifiers are accepted on allow / deny; categories are also available through the BotCategory enum.

Individual bots

The bot list contains a list of known bots that Arcjet can identify.

For example:

JavaScript / TypeScript

detectBot({
   mode: "LIVE", // will block requests. Use "DRY_RUN" to log only
   // Block all bots except specific Google and Bing crawlers, and curl
   allow: [
      "GOOGLE_CRAWLER",
      "GOOGLE_CRAWLER_NEWS",
      "BING_CRAWLER",
      "CURL",
   ],
}),

Python

detect_bot(
    mode=Mode.LIVE,  # Blocks requests. Use Mode.DRY_RUN to log only
    # Block all bots except specific Google and Bing crawlers, and curl
    allow=[
        "GOOGLE_CRAWLER",
        "GOOGLE_CRAWLER_NEWS",
        "BING_CRAWLER",
        "CURL",
    ],
)

Bot categories

We provide the following categories:

• CATEGORY:ACADEMIC: Scrape data for research purposes
• CATEGORY:ADVERTISING: Scrape data for advertising and marketing purposes
• CATEGORY:AI: Scrape data for AI and LLM purposes
• CATEGORY:AMAZON: Scrape data for Amazon products and services
• CATEGORY:ARCHIVE: Scrape data for archival purposes
• CATEGORY:BOTNET: Automated behavior for malicious botnets
• CATEGORY:FEEDFETCHER: Request data for RSS and other feeds
• CATEGORY:GOOGLE: Scrape data for Google products and services
• CATEGORY:META: Scrape data for Meta/Facebook products and services
• CATEGORY:MICROSOFT: Scrape data for Microsoft products and services
• CATEGORY:MONITOR: Interact for monitoring purposes
• CATEGORY:OPTIMIZER: Interact for optimization purposes
• CATEGORY:PREVIEW: Request data for image and URL previews
• CATEGORY:PROGRAMMATIC: Interact through programming language libraries
• CATEGORY:SEARCH_ENGINE: Index data for search engines
• CATEGORY:SLACK: Scrape data for Slack products and services
• CATEGORY:SOCIAL: Scrape data for social media products and services
• CATEGORY:TOOL: Interact through command line and GUI tools
• CATEGORY:UNKNOWN: Known bots that cannot be classified
• CATEGORY:VERCEL: Scrape data for Vercel products and services
• CATEGORY:WEBHOOK: Automated HTTP notifications for events, such as Stripe payments
• CATEGORY:YAHOO: Scrape data for Yahoo products and services

The bot list contains a list of categories and which bots are in each category.

JavaScript / TypeScript

detectBot({
   mode: "LIVE", // will block requests. Use "DRY_RUN" to log only
   // Block all bots except search engines and curl
   allow: [
      "CATEGORY:SEARCH_ENGINE", // Google, Bing, etc
      "CURL", // You can allow specific bots in addition to categories
   ],
}),

Python

from arcjet import BotCategory, detect_bot, Mode

detect_bot(
    mode=Mode.LIVE,  # Blocks requests. Use Mode.DRY_RUN to log only
    # Block all bots except search engines and curl
    allow=[
        BotCategory.SEARCH_ENGINE,  # Google, Bing, etc
        "CURL",  # You can allow specific bots in addition to categories
    ],
)

The Python SDK also accepts the underlying string identifiers (for example, "CATEGORY:SEARCH_ENGINE") if you prefer not to import the enum.

Only configured categories are checked for performance reasons. Each detected bot must be compared to a category, so the worst case performance is count(detectedBot) * count(configuredCategories).

We’re continuously evaluating bots to decide whether any need reclassifying. If we determine enough bots exist for a new category, we’ll consider adding new ones. If you need a specific category, open an issue on the arcjet/well-known-bots repository.

Custom bots

To block a particular bot that is not on our list, you can use Arcjet Filters. For how to block custom bots, see the Malicious traffic blueprint.

Detection

Arcjet provides additional bot verification using IP analysis, which can help if you are under attack from bots pretending to be good bots, such as clients pretending to be Google (whom you usually want to allow).

Note

Requests without User-Agent headers might not be identified as any particular bot and could be marked as an errored result. Our recommendation is to block requests without User-Agent headers because most legitimate clients send this header.

See an example of how to do this.

Known bots structure

The identifiers on the bot list are generated from a collection of known bots which includes details of their owner and any variations.

We welcome contributions to the arcjet/well-known-bots repository, whether you’re adding new bots or updating detection patterns. Once merged, the updates are included in the next SDK release. Because bot detection is handled within the Arcjet WebAssembly module bundled with the SDK, new patterns must be compiled into the module as part of the release process.

For instructions on how to contribute to our bot detection, read the repository’s README.md.

Unknown bots

If a bot is detected but cannot be identified as a known bot, Arcjet labels it UNKNOWN_BOT. This is separate from the CATEGORY:UNKNOWN category, which is for bots that cannot be classified into any category but can still be identified as a specific bot. You can see a list of these named, but unclassified bots in the bot list.

Detections returned as UNKNOWN_BOT happen if the bot is new or hides itself. It’s a bot with no name. Arcjet uses various techniques to detect these bots, including analyzing request patterns and tracking IP addresses.

If you configure an allow rule and do not include UNKNOWN_BOT, then Arcjet blocks detected bots that it cannot identify. This is the default behavior to protect against new and rapidly evolving bots.

If you want to be more permissive and only block known, named bots, you can include UNKNOWN_BOT in your allow rules.

Discussion