Skip to content

Block malicious bots, scrapers, and spoofed crawlers

A malicious bot is an automated client that requests your site to take something or to abuse it, such as a scraper that copies your content, a tool that tries stolen passwords, or a script that creates fake accounts. Arcjet blocks malicious bots by denying every bot that your site doesn’t need, catching clients that pretend to be a good bot, and flagging automated browsers and risky networks.

No single check catches every bot. The sections on this page add one layer each, and the last section combines them on a signup route. The examples use @arcjet/node, FastAPI, and net/http. The same rules work in Next.js and the other JavaScript SDKs.

The following kinds of bots cause most of the harm:

  • Scrapers copy content, prices, and listings, and add load to your servers. Many send a browser User-Agent header and rotate through cloud and proxy IP addresses.
  • Credential stuffing tools try stolen username and password pairs against your login form, often from many IP addresses at a low rate each.
  • Signup and form bots create fake accounts to abuse free tiers, trials, and referral programs, or fill contact forms with spam.
  • Spoofed crawlers send Googlebot’s or another search engine’s User-Agent header, because many sites let search engines through.

Start from deny by default. Use an allow list that names the bots and categories that your site needs, and Arcjet blocks every other detected bot, including bots that it can’t name (UNKNOWN_BOT). Most sites need search engines, link previews in chat and social apps, and their own uptime monitors.

An allowed bot can still be a fake. Arcjet verifies bots that claim to be from an operator that publishes its IP addresses or hostnames, such as Googlebot, and marks a request that fails the check as spoofed. The allow list still allows a spoofed bot, so the decision alone doesn’t stop it. Also deny the request when isSpoofedBot in JavaScript, is_spoofed_bot in Python, or decision.IsSpoofedBot() in Go returns true:

index.ts
import arcjet, { detectBot } from "@arcjet/node";
import { isSpoofedBot } from "@arcjet/inspect";
import http from "node:http";
const aj = arcjet({
key: process.env.ARCJET_KEY!,
rules: [
detectBot({
mode: "LIVE", // Blocks requests. Use "DRY_RUN" to log only
// Block every detected bot except the ones this site needs
allow: [
"CATEGORY:SEARCH_ENGINE", // Google, Bing, etc
"CATEGORY:PREVIEW", // Link previews in chat apps
"CATEGORY:MONITOR", // Uptime monitors
],
}),
],
});
const server = http.createServer(async (req, res) => {
const decision = await aj.protect(req);
// Deny blocked bots and allowed bots that failed verification
if (decision.isDenied() || decision.results.some(isSpoofedBot)) {
res.writeHead(403, { "Content-Type": "application/json" });
res.end(JSON.stringify({ error: "Forbidden" }));
return;
}
res.writeHead(200, { "Content-Type": "application/json" });
res.end(JSON.stringify({ message: "Hello world" }));
});
server.listen(3000);

If you rely on AI agents, add CATEGORY:AI_AGENT to the list. For more information, see Detect AI agents. For every category, see Bot categories.

Catch headless browsers and browser automation

Section titled “Catch headless browsers and browser automation”

Many scrapers and signup bots run a real browser, such as headless Chrome driven by Playwright or Puppeteer. They send a normal User-Agent header, so the bots list doesn’t name them.

Advanced signals catch these bots in the browser. A script on your pages runs checks that detect automation, and stores the result in the aj_signals cookie. On the next request, detectBot reads the cookie and denies a browser that failed the checks, with an identity of kind AUTOMATION and FINGERPRINT evidence.

A bot that never runs your pages’ JavaScript never gets the cookie. A missing cookie doesn’t make detectBot deny a request, so on routes that only a browser reaches, such as a form POST handler, require the cookie with a filter rule:

rules: [
// Deny requests without the advanced signals cookie
filter({
mode: "LIVE",
deny: ['len(http.request.cookie["aj_signals"]) eq 0'],
}),
// Deny every detected bot, including failed advanced signals checks
detectBot({ mode: "LIVE", allow: [] }),
],

Load the signals script on every page that leads to the route, so that real visitors have the cookie before they submit the form. For setup, see Advanced signals setup.

Bots run where computing is cheap and hard to trace: hosting providers, VPNs, open proxies, and Tor. Each decision includes Arcjet’s analysis of the client IP address, so you can treat these networks differently from home and mobile connections:

  • In JavaScript, use decision.ip.isHosting(), isVpn(), isProxy(), and isTor(), and read the threat assessment from decision.ip.threat.
  • In Python, use decision.ip.is_hosting(), is_vpn(), is_proxy(), and is_tor(), and read the assessment from decision.ip_details.threat.
  • In Go, read decision.IP.IsHosting, IsVPN, IsProxy, and IsTor, and the assessment from decision.IP.Threat.

The threat assessment is present only when Arcjet has one. It includes a risk level from none to critical, a reputation such as malicious or suspicious, and the abuse activities that Arcjet has seen from the IP address, such as brute_force. Skip a denial when isSafe is true, because that marks trusted infrastructure.

People also use VPNs and hosting networks, so a network type alone isn’t proof of abuse. Block the strongest signals, such as Tor or a high or critical risk level, on sensitive routes, and challenge the weaker ones, for example with email verification. The example in the next section does both. For every field and value, see IP threat intelligence.

Bots that pass every identity check can still send too many requests or attack your application. Add a rate limit to cap how often one client can call a route, and Shield to block common attacks, such as SQL injection.

The following example protects a signup route with every layer on this page. It blocks attacks, rate limits each IP address, requires the advanced signals cookie, blocks every detected bot and spoofed bot, blocks Tor, open proxies, and high-risk IP addresses, and asks hosting and VPN traffic to verify an email address:

index.ts
import arcjet, { detectBot, filter, shield, slidingWindow } from "@arcjet/node";
import { isSpoofedBot } from "@arcjet/inspect";
import http from "node:http";
const aj = arcjet({
key: process.env.ARCJET_KEY!,
rules: [
// Block common attacks, such as SQL injection
shield({ mode: "LIVE" }),
// Allow 5 signup attempts per IP address every 10 minutes
slidingWindow({ mode: "LIVE", interval: "10m", max: 5 }),
// Require the advanced signals cookie
filter({
mode: "LIVE",
deny: ['len(http.request.cookie["aj_signals"]) eq 0'],
}),
// Block every detected bot and failed advanced signals checks
detectBot({ mode: "LIVE", allow: [] }),
],
});
const server = http.createServer(async (req, res) => {
if (req.method !== "POST" || req.url !== "/signup") {
res.writeHead(404);
res.end();
return;
}
const decision = await aj.protect(req);
if (decision.isDenied()) {
const status = decision.reason.isRateLimit() ? 429 : 403;
res.writeHead(status, { "Content-Type": "application/json" });
res.end(JSON.stringify({ error: "Request blocked" }));
return;
}
const { ip } = decision;
const threat = ip.threat;
const highRisk =
threat &&
!threat.isSafe &&
(threat.riskLevel === "high" || threat.riskLevel === "critical");
// Block spoofed bots, Tor, open proxies, and high-risk IP addresses
if (
decision.results.some(isSpoofedBot) ||
ip.isTor() ||
ip.isProxy() ||
highRisk
) {
res.writeHead(403, { "Content-Type": "application/json" });
res.end(JSON.stringify({ error: "Request blocked" }));
return;
}
// Challenge hosting and VPN traffic instead of blocking it
if (ip.isHosting() || ip.isVpn() || threat?.reputation === "suspicious") {
res.writeHead(202, { "Content-Type": "application/json" });
res.end(JSON.stringify({ next: "verify-email" }));
return;
}
res.writeHead(201, { "Content-Type": "application/json" });
res.end(JSON.stringify({ message: "Account created" }));
});
server.listen(3000);

Start new rules in DRY_RUN mode and review the results in the Arcjet Console before you block traffic. For signup forms, also consider signup form protection, which adds email validation.

Add a detectBot rule with an allow list that names only the bots that your site needs, such as CATEGORY:SEARCH_ENGINE. Arcjet then blocks every other detected bot, including bots that it can’t name. Add advanced signals, IP threat intelligence, and rate limiting to catch bots that hide their identity.

How do I stop bots from spoofing Googlebot?

Section titled “How do I stop bots from spoofing Googlebot?”

Allow Googlebot, then deny requests where isSpoofedBot in JavaScript, is_spoofed_bot in Python, or decision.IsSpoofedBot() in Go returns true. Arcjet checks a Googlebot claim against Google’s published IP address ranges and reverse DNS, and marks a request that fails the check as spoofed.

Arcjet verifies Googlebot for you when your allow list includes it, for example through CATEGORY:SEARCH_ENGINE. Check isVerifiedBot in JavaScript, is_verified_bot in Python, or decision.IsVerifiedBot() in Go to see whether Arcjet confirmed the bot. For more information, see Bot verification.

Use advanced signals. A script on your pages runs checks in the browser that detect automation, such as headless Chrome driven by Playwright or Puppeteer, and detectBot denies a browser that fails them. Require the aj_signals cookie with a filter rule to also block clients that never run the script.

Arcjet helps reduce credential stuffing on a login route when you combine a detectBot rule, advanced signals, a rate limit, and IP threat intelligence, which reports IP addresses seen in brute force activity. No bot detection is complete, so keep your own account protections, such as multi-factor authentication.

Should I block all hosting and VPN traffic?

Section titled “Should I block all hosting and VPN traffic?”

Not on public pages, because real people use VPNs and some legitimate clients run on hosting providers. On sensitive routes, such as signup and login, block Tor and high-risk IP addresses, and challenge hosting and VPN traffic instead of blocking it.

Discussion