Skip to content

Next.js bot protection reference

Arcjet bot detection allows you to manage traffic by automated clients and bots.

Configuration

Bot detection is configured by specifying the bot types you wish to block and optional user agent patterns to add or remove from the bot detection list.

The configuration definition is:

type BotOptions = {
mode?: "LIVE" | "DRY_RUN";
block?: BotType[];
patterns?: {
add?: { [key: string]: BotType };
remove?: string[];
};
};

The arcjet client is configured with one or more detectBot rules which take one or many BotOptions.

Which bots to block?

Which bot types to block is configured by listing one or more types in the block configuration block. The types are listed on the bot types page.

Adding bot detection rules

You can add additional bot detection rules to the patterns add configuration property. Each rule is a regular expression that matches the user agent of the bot plus a label to indicate what type of bot it is from ArcjetBotTypes described in the list of bot types. The following example adds a rule to detect Googlebot as a LikelyAutomated bot.

/middleware.ts
import arcjet, { createMiddleware, detectBot } from "@arcjet/next";
export const config = {
// matcher tells Next.js which routes to run the middleware on.
// This runs the middleware on all routes except for static assets.
matcher: ["/((?!_next/static|_next/image|favicon.ico).*)"],
};
const aj = arcjet({
key: process.env.ARCJET_KEY,
rules: [
detectBot({
mode: "LIVE",
block: [
// Only block clients we're sure are automated bots
"AUTOMATED",
],
patterns: {
add: {
// Marks Googlebot as a likely automated bot so it will not be blocked
"Googlebot\\/": "LIKELY_AUTOMATED",
},
},
}),
],
});
// Pass any existing middleware with the optional existingMiddleware prop
export default createMiddleware(aj);

Removing bot detection rules

Arcjet includes a set of default matching rules to detect common bots. You can remove any of these rules by listing them in the patterns remove configuration property:

/middleware.ts
import arcjet, { createMiddleware, detectBot } from "@arcjet/next";
export const config = {
// matcher tells Next.js which routes to run the middleware on.
// This runs the middleware on all routes except for static assets.
matcher: ["/((?!_next/static|_next/image|favicon.ico).*)"],
};
const aj = arcjet({
key: process.env.ARCJET_KEY,
rules: [
detectBot({
mode: "LIVE",
block: [
// Only block clients we're sure are automated bots
"AUTOMATED",
],
patterns: {
remove: [
// Removes the datadog agent from the list of bots so it will be
// considered as ArcjetBotType.LIKELY_NOT_A_BOT
"datadog agent",
// Also allow curl clients to pass through. Matches a user agent
// string with the word "curl" in it
"^curl",
],
},
}),
],
});
// Pass any existing middleware with the optional existingMiddleware prop
export default createMiddleware(aj);

Per route vs middleware

Bot protection rules can be configured in two ways:

  • Per API route: The rule is defined in the API route itself. This allows you to configure the rule alongside the code it is protecting which is useful if you want to use the decision to add context to your own code. However, it means rules are not located in a single place.
  • Middleware: The rule is defined in the middleware. This allows you to configure rules in a single place or apply them globally to all routes, but it means the rules are not located alongside the code they are protecting.

Per route

This configures bot protection on a single route.

/app/api/arcjet/route.ts
import arcjet, { detectBot } from "@arcjet/next";
import { NextResponse } from "next/server";
const aj = arcjet({
key: process.env.ARCJET_KEY,
rules: [
detectBot({
mode: "LIVE",
block: ["AUTOMATED", "LIKELY_AUTOMATED"],
}),
],
});
export async function GET(req: Request) {
const decision = await aj.protect(req);
if (decision.isDenied() && decision.reason.isBot()) {
return NextResponse.json(
{
error: "You are a bot!",
// Useful for debugging, but don't return it to the client in
// production
//reason: decision.reason,
},
{ status: 403 },
);
}
return NextResponse.json({
message: "Hello world",
});
}

Middleware

This will run on every request to your Next.js app, except for static assets (configured in the matcher - see the Next.js docs for details).

Create a file called middleware.ts in your project root (at the same level as pages or app or inside src):

/middleware.ts
import arcjet, { createMiddleware, detectBot } from "@arcjet/next";
export const config = {
// matcher tells Next.js which routes to run the middleware on.
// This runs the middleware on all routes except for static assets.
matcher: ["/((?!_next/static|_next/image|favicon.ico).*)"],
};
const aj = arcjet({
key: process.env.ARCJET_KEY!, // Get your site key from https://app.arcjet.com
rules: [
detectBot({
mode: "LIVE", // will block requests. Use "DRY_RUN" to log only
block: ["AUTOMATED"], // blocks all automated clients
}),
],
});
// Pass any existing middleware with the optional existingMiddleware prop
export default createMiddleware(aj);

Avoiding double protection with middleware

If you use Arcjet in middleware and individual routes, you need to be careful that Arcjet is not running multiple times per request. This can be avoided by excluding the API route from the middleware matcher.

For example, if you already have a bot detection rule defined in the API route at /api/hello, you can exclude it from the middleware by specifying a matcher in /middleware.ts:

/middleware.ts
import arcjet, { createMiddleware, detectBot } from "@arcjet/next";
export const config = {
// The matcher prevents the middleware executing on static assets and the
// /api/hello API route because you already installed Arcjet directly
matcher: ["/((?!_next/static|_next/image|favicon.ico|api/hello).*)"],
};
const aj = arcjet({
key: process.env.ARCJET_KEY,
rules: [
detectBot({
mode: "LIVE",
block: ["AUTOMATED", "LIKELY_AUTOMATED"],
}),
],
});
// Pass any existing middleware with the optional existingMiddleware prop
export default createMiddleware(aj);

Decision

The quick start example will deny requests that match the bot detection rules, immediately returning a response to the client using Next.js middleware.

Arcjet also provides a single protect function that is used to execute your protection rules. This requires a request property which is the request context as passed to the request handler.

This function returns a Promise that resolves to an ArcjetDecision object. This contains the following properties:

  • id (string) - The unique ID for the request. This can be used to look up the request in the Arcjet dashboard. It is prefixed with req_ for decisions involving the Arcjet cloud API. For decisions taken locally, the prefix is lreq_.
  • conclusion (ArcjetConclusion) - The final conclusion based on evaluating each of the configured rules. If you wish to accept Arcjet’s recommended action based on the configured rules then you can use this property.
  • reason (ArcjetReason) - An object containing more detailed information about the conclusion.
  • results (ArcjetRuleResult[]) - An array of ArcjetRuleResult objects containing the results of each rule that was executed.
  • ip (ArcjetIpDetails) - An object containing Arcjet’s analysis of the client IP address. See IP analysis in the SDK reference for more information.

See the SDK reference for more details about the rule results.

You check if a deny conclusion has been returned by a bot protection rule by using decision.isDenied() and decision.reason.isBot() respectively.

You can iterate through the results and check whether a bot protection rule was applied:

for (const result of decision.results) {
console.log("Rule Result", result);
}

This example will log the full result as well as the bot protection rule:

Create a new API route at /app/api/route/hello.ts:

/app/api/route/hello.ts
import arcjet, { fixedWindow, detectBot } from "@arcjet/next";
import { NextResponse } from "next/server";
const aj = arcjet({
key: process.env.ARCJET_KEY,
rules: [
fixedWindow({
mode: "LIVE",
// Limiting by ip.src is the default if not specified
//characteristics: ["ip.src"],
window: "1h",
max: 60,
}),
detectBot({
mode: "LIVE",
block: ["AUTOMATED", "LIKELY_AUTOMATED"],
}),
],
});
export async function POST(req: Request) {
const decision = await aj.protect(req);
for (const result of decision.results) {
console.log("Rule Result", result);
if (result.reason.isRateLimit()) {
console.log("Rate limit rule", result);
}
if (result.reason.isBot()) {
console.log("Bot protection rule", result);
}
}
if (decision.isDenied()) {
return NextResponse.json({ error: "Forbidden" }, { status: 403 });
}
return NextResponse.json({
message: "Hello world",
});
}

Bot type

Arcjet also returns more information about the bot type of the client we think made the request:

Create a new API route at /app/api/route/hello.ts:

/app/api/route/hello.ts
import arcjet, { detectBot } from "@arcjet/next";
import { NextResponse } from "next/server";
const aj = arcjet({
key: process.env.ARCJET_KEY,
rules: [
detectBot({
mode: "LIVE",
block: ["AUTOMATED", "LIKELY_AUTOMATED"],
}),
],
});
export async function POST(req: Request) {
const decision = await aj.protect(req);
if (decision.isDenied() && decision.reason.isBot()) {
return NextResponse.json(
{
error: "You are a bot!",
// Useful for debugging, but don't return these to the client in
// production
botType: decision.reason.botType,
botScore: decision.reason.botScore,
ipHosting: decision.reason.ipHosting,
ipVpn: decision.reason.ipVpn,
ipProxy: decision.reason.ipProxy,
ipTor: decision.reason.ipTor,
ipRelay: decision.reason.ipRelay,
userAgentMatch: decision.reason.ipRelay,
},
{ status: 403 },
);
}
return NextResponse.json({
message: "Hello world",
});
}

Error handling

Arcjet is designed to fail open so that a service issue or misconfiguration does not block all requests. The SDK will also time out and fail open after 500ms when NODE_ENV is production and 1000ms otherwise. However, in most cases, the response time will be less than 20-30ms.

If there is an error condition, Arcjet will return an ERROR type and you can check the reason property for more information, like accessing decision.reason.message.

/app/api/hello/route.ts
import arcjet, { detectBot } from "@arcjet/next";
import { NextResponse } from "next/server";
const aj = arcjet({
key: process.env.ARCJET_KEY,
rules: [
detectBot({
mode: "LIVE",
block: ["AUTOMATED", "LIKELY_AUTOMATED"],
}),
],
});
export async function GET(req: Request) {
const decision = await aj.protect(req);
if (decision.isErrored()) {
// Fail open by logging the error and continuing
console.warn("Arcjet error", decision.reason.message);
// You could also fail closed here for very sensitive routes
//return NextResponse.json({ error: "Service unavailable" }, { status: 503 });
}
if (decision.isDenied()) {
return NextResponse.json(
{
error: "You are a bot!",
// Useful for debugging, but don't return it to the client in
// production
//reason: decision.reason,
},
{
status: 403,
},
);
}
return NextResponse.json({
message: "Hello world",
});
}

Examples

Protecting a page

You can protect a Next.js page from bots by calling the Arcjet SDK from within the page loader:

Protecting an app router page within the handler itself is not currently supported, but you can set up a matcher on the middleware instead:

/middleware.ts
import arcjet, { createMiddleware, detectBot } from "@arcjet/next";
export const config = {
// The matcher runs just on the /hello pages route
matcher: ["/hello"],
};
const aj = arcjet({
key: process.env.ARCJET_KEY,
rules: [
detectBot({
mode: "LIVE",
block: ["AUTOMATED", "LIKELY_AUTOMATED"],
}),
],
});
// Pass any existing middleware with the optional existingMiddleware prop
export default createMiddleware(aj);

Wrap existing handler

All the examples on this page show how you can inspect the decision to control what to do next. However, if you just wish to send a generic 403 Forbidden response you can delegate this to Arcjet by wrapping your handler withArcjet.

For both the Node or Edge runtime:

/app/api/hello/route.ts
import arcjet, { detectBot, withArcjet } from "@arcjet/next";
import { NextResponse } from "next/server";
const aj = arcjet({
key: process.env.ARCJET_KEY,
rules: [
detectBot({
mode: "LIVE",
block: ["AUTOMATED", "LIKELY_AUTOMATED"],
}),
],
});
export const GET = withArcjet(aj, async (req: Request) => {
return NextResponse.json({
message: "Hello world",
});
});

Edge Functions

Arcjet works in Edge Functions and with the Edge Runtime.

/app/api/hello/route.ts
import arcjet, { detectBot } from "@arcjet/next";
import { NextRequest, NextResponse } from "next/server";
export const config = {
runtime: "edge",
};
const aj = arcjet({
key: process.env.ARCJET_KEY,
rules: [
detectBot({
mode: "LIVE",
block: ["AUTOMATED", "LIKELY_AUTOMATED"],
}),
],
});
export default async function handler(req: NextRequest, res: NextResponse) {
const decision = await aj.protect(req);
if (decision.isDenied()) {
return NextResponse.json(
{
error: "You are a bot!",
// Useful for debugging, but don't return it to the client in production
//reason: decision.reason,
},
{
status: 403,
},
);
}
return NextResponse.json({
message: "Hello world",
});
}