Bot protection reference

Arcjet bot detection lets you manage traffic by automated clients and bots.

Configuration

Bot detection is configured by allowing or denying a subset of bots. You must pass exactly one of allow or deny. The lists are mutually exclusive. With allow, Arcjet denies any detected bot that the list does not name. With deny, Arcjet allows any detected bot that the list does not name.

An empty allow list is valid and blocks every detected bot. In Python, detect_bot and the BotDetection dataclass raise ValueError if you pass neither list or both lists.

You can use only one of the following configuration definitions:

JavaScript / TypeScript

type BotOptionsAllow = {
  mode?: "LIVE" | "DRY_RUN";
  allow: Array<ArcjetWellKnownBot | ArcjetBotCategory>;
};

type BotOptionsDeny = {
  mode?: "LIVE" | "DRY_RUN";
  deny: Array<ArcjetWellKnownBot | ArcjetBotCategory>;
};

The arcjet client is configured with one or more detectBot rules which take one or many BotOptions.

Python

# Signature for arcjet.detect_bot
# Pass exactly one of `allow` or `deny`. Passing neither or both raises
# ValueError. An empty allow=[] blocks every detected bot.
def detect_bot(
    *,
    # Required. Mode.LIVE blocks requests; Mode.DRY_RUN logs only.
    mode: Mode,
    # Bots to permit. All other detected bots are denied.
    allow: Sequence[str | BotCategory] | None = None,
    # Bots to block. All other detected bots are allowed.
    deny: Sequence[str | BotCategory] | None = None,
) -> BotDetection: ...

The arcjet (or arcjet_sync) client is configured with one or more detect_bot rules. Pass BotCategory.<NAME> enum values for categories and string identifiers like "CURL" for specific bots.

Declaration order does not control which LIVE deny you see. The JS, Go, and Python SDKs sort local Protect rules in the same order: Sensitive Info, Filter, Shield, rate limiting, Bot, Email, then Prompt Injection. Rules with the same priority keep their declaration order. For details about examining the results, see Decision. For the Python priority table, see Multiple rules in the Python SDK reference.

Allow specific bots

Most applications want to block almost all bots. However, it is common to allow some bots to access your system, such as bots for search indexing or API access from the command line.

When allowing specific bots we recommend that you also check the verification status after an allow decision is returned to ensure that the bots are who they say they are.

Configure this behavior with an allow list drawn from the full list of bots, from bot categories, or from both.

Deny specific bots

Some applications may only want to block a small subset of bots, while allowing the majority continued access. This may be due to many reasons, such as misconfigured or high-traffic bots.

Configure this behavior with a deny list drawn from the full list of bots, from bot categories, or from both.

Decision

The quick start example denies requests that match the bot detection rules, immediately returning a response to the client.

Arcjet provides a single protect function that is used to execute your protection rules. This requires a request argument which is the request context as passed to the request handler.

This function returns a Promise that resolves to an ArcjetDecision object. This contains the following properties:

• id (string) – The unique ID for the request. This can be used to look up the request in the Arcjet dashboard. It is prefixed with req_ for decisions involving the Arcjet cloud API. For decisions taken locally, the prefix is lreq_.
• conclusion (ArcjetConclusion) – The final conclusion based on evaluating each of the configured rules. If you wish to accept Arcjet’s recommended action based on the configured rules then you can use this property.
• reason (ArcjetReason) – An object containing more detailed information about the conclusion.
• results (ArcjetRuleResult[]) – An array of ArcjetRuleResult objects containing the results of each rule that was executed.
• ip (ArcjetIpDetails) – An object containing Arcjet’s analysis of the client IP address. For more information, see the SDK reference.

To check whether a bot protection rule returned a deny conclusion, use decision.isDenied() and decision.reason.isBot() (JS) / decision.is_denied() and decision.reason_v2.type == "BOT" (Python).

You can iterate through the results and check whether a bot protection rule was applied:

JavaScript / TypeScript

for (const result of decision.results) {
  console.log("Rule Result", result);
}

Python

for result in decision.results:
    print("Rule Result", result)

Identified bots

The decision also contains all of the identified bots and matched categories detected from the request. A request may be identified as zero, one, or more bots/categories-all of which are available on the decision.allowed and decision.denied properties.

Error handling

Arcjet is designed to fail open so that a service issue or misconfiguration does not block all requests. The SDK also times out and fails open after 2000 ms by default. However, in most cases, the response time is less than 20 ms to 30 ms.

If there is an error condition when processing the rule, Arcjet returns an ERROR result for that rule and you can check the message property on the rule’s error result for more information.

If all other rules that were run returned an ALLOW result, then the final Arcjet conclusion is ERROR.

Note

Requests without User-Agent headers might not be identified as any particular bot and could be marked as an errored result. Our recommendation is to block requests without User-Agent headers because most legitimate clients send this header.

See an example of how to do this.

Filter categories

All categories are also provided as enumerations, which allows for programmatic access. For example, you may want to allow most of CATEGORY:GOOGLE except their “advertising quality” bot.

Bot verification

Requests analyzed by Arcjet include automatic bot verification. For allow rules, Arcjet verifies the authenticity of detected bots by checking IP data and performing reverse DNS lookups.

This helps protect against spoofed bots where clients pretend to be someone else.

Note

Because verification is only helpful when you want to specifically allow certain bots, any deny rules bypass verification and block bots matching the deny list.

Example: Allowing verified bots

Well-behaved bots, such as search engine indexers, are often desirable traffic. The companies that operate these bots make them verifiable so application developers can choose to avoid additional signals about the request.

For example, when a request claims to be GoogleBot, Arcjet checks whether the IP truly belongs to Google. You can check the verification status in your code and take actions based on the results, such as allowing all verified bots.

JavaScript / TypeScript

import { isVerifiedBot } from "@arcjet/inspect";

// ...
const aj = arcjet({
  // ...
  rules: [
    detectBot({
      mode: "LIVE",
      allow: ["CATEGORY:SEARCH_ENGINE"],
    }),
  ],
});

// ...
const decision = await aj.protect(req);
// ...

// Ignore other signals for verified search engine bots
if (decision.results.some(isVerifiedBot)) {
  return new Response("Hello Bot!");
}

// Leverage all Arcjet signals
if (decision.isDenied()) {
  return new Response(null, { status: 403 });
}

Python

from arcjet import Mode, arcjet, detect_bot, is_verified_bot

# ...
aj = arcjet(
    key=os.environ["ARCJET_KEY"],
    rules=[
        detect_bot(mode=Mode.LIVE, allow=["CATEGORY:SEARCH_ENGINE"]),
    ],
)

# ...
decision = await aj.protect(request)
# ...

# Ignore other signals for verified search engine bots
if any(is_verified_bot(r) for r in decision.results):
    return JSONResponse({"message": "Hello Bot!"})

# Leverage all Arcjet signals
if decision.is_denied():
    return JSONResponse(None, status_code=403)

Check for spoofed bots

This checks whether the bot is spoofed. You would usually return a 403 or similar response to block the request.

JavaScript / TypeScript

for (const { reason } of decision.results) {
  if (reason.isBot() && reason.isSpoofed()) {
    console.log("Detected spoofed bot", reason.spoofed);
    // Return a 403 or similar response
  }
}

Python

from arcjet import is_spoofed_bot

# Use the is_spoofed_bot() helper exported by arcjet:
if any(is_spoofed_bot(r) for r in decision.results):
    print("Detected spoofed bot")
    # Return a 403 or similar response

# Or inspect each rule result directly:
for result in decision.results:
    if result.reason_v2.type == "BOT" and result.reason_v2.spoofed:
        print("Detected spoofed bot")
        # Return a 403 or similar response

Check bot verification

This checks whether the bot is verified.

JavaScript / TypeScript

for (const { reason } of decision.results) {
  if (reason.isBot() && reason.isVerified()) {
    console.log("Verified bot", reason.verified);
    // Allow the request
  }
}

Python

from arcjet import is_verified_bot

# Use the is_verified_bot() helper exported by arcjet:
if any(is_verified_bot(r) for r in decision.results):
    print("Verified bot")
    # Allow the request

# Or inspect each rule result directly:
for result in decision.results:
    if result.reason_v2.type == "BOT" and result.reason_v2.verified:
        print("Verified bot")
        # Allow the request

User-Agent header

Requests without User-Agent headers cannot be identified as any particular bot and are marked as an errored result.

Most legitimate clients send it because HTTP/1.1 (RFC 7231) says it should be sent. You can choose to block such requests with Arcjet Filters.

JavaScript / TypeScript

filter({
  // This will deny any traffic that has no user agent:
  deny: ['len(http.request.headers["user-agent"]) eq 0'],
  // Block requests with `LIVE`, use `DRY_RUN` to log only.
  mode: "LIVE",
}),

Python

filter_request(
    # This will deny any traffic that has no user agent:
    deny=['len(http.request.headers["user-agent"]) eq 0'],
    # Block requests with `LIVE`, use `Mode.DRY_RUN` to log only.
    mode=Mode.LIVE,
)

An alternative approach is to check the rule results after the decision is made:

JavaScript / TypeScript

Use the isMissingUserAgent helper from @arcjet/inspect:

import { isMissingUserAgent } from "@arcjet/inspect";

if (decision.results.some(isMissingUserAgent)) {
  log.warn(
    { error: decision.reason.message },
    "request missing required user-agent header",
  );
  // You could return a 400 Bad request error here
  // Next.js example:
  // return NextResponse.json({ error: "Bad request" }, { status: 400 });
  // Node.js example:
  // res.writeHead(400, { "Content-Type": "application/json" });
  // res.end(JSON.stringify({ error: "Bad request" }));
}

Python

Use the is_missing_user_agent helper from arcjet:

from arcjet import is_missing_user_agent

if any(is_missing_user_agent(r) for r in decision.results):
    logger.warning("request missing required user-agent header")
    # You could return a 400 Bad request error here
    return JSONResponse({"error": "Bad request"}, status_code=400)

Testing

Arcjet runs the same in any environment, including locally and in CI. You can use the mode set to DRY_RUN to log the results of rule execution without blocking any requests.

We have an example test framework you can use to automatically test your rules. Arcjet can also be triggered based using a sample of your traffic.

For details, see the Testing section of the docs.

Discussion