Skip to content
Arcjet Docs

AI prompt injection detection

Prompt injection attacks trick AI models into ignoring their instructions. Users paste in jailbreaks such as “DAN” prompts, role-play escapes, or instruction overrides designed to bypass your system prompt, extract restricted information, or make your AI behave in unintended ways.

Arcjet prompt injection detection evaluates each incoming message for injection patterns inside your application before it reaches the AI provider. Detected attacks are blocked before the AI call is made, protecting both your application behavior and your AI budget.

Get started

Section titled “Get started”

Protect a production chat endpoint

Section titled “Protect a production chat endpoint”

A production chat endpoint needs more than one guardrail. Some requests contain hostile instructions designed to override your system prompt. Others may be legitimate user requests that still contain sensitive data you do not want entering model context. And like any other public route, AI endpoints still need protection from common web attacks.

Combining Arcjet rules gives you layered enforcement before the model runs:

  • Shield blocks common web attacks against the endpoint
  • Prompt injection detection catches hostile instructions before inference
  • Sensitive information detection prevents PII from entering model context

The following example uses the Vercel AI SDK (JS) / LangChain (Python):

import { openai } from "@ai-sdk/openai";
import arcjet, {
  detectPromptInjection,
  sensitiveInfo,
  shield,
} from "@arcjet/next";
import type { UIMessage } from "ai";
import { convertToModelMessages, isTextUIPart, streamText } from "ai";


const aj = arcjet({
  key: process.env.ARCJET_KEY!, // Get your site key from https://app.arcjet.com
  rules: [
    // Shield protects against common web attacks e.g. SQL injection
    shield({ mode: "LIVE" }),


    // Detect prompt injection attacks before they reach your AI model
    detectPromptInjection({
      mode: "LIVE",
    }),


    // Block sensitive data from entering model context
    sensitiveInfo({
      mode: "LIVE",
      // Block PII types that should never appear in AI prompts.
      // Remove types your app legitimately handles (e.g. EMAIL for a support bot).
      deny: ["CREDIT_CARD_NUMBER", "EMAIL"],
    }),
  ],
});


export async function POST(req: Request) {
  const { messages }: { messages: UIMessage[] } = await req.json();


  // Check the most recent user message.
  // Pass the full conversation if you want to scan all messages.
  const lastMessage: string = (messages.at(-1)?.parts ?? [])
    .filter(isTextUIPart)
    .map((p) => p.text)
    .join(" ");


  const decision = await aj.protect(req, {
    detectPromptInjectionMessage: lastMessage,
    sensitiveInfoValue: lastMessage,
  });


  if (decision.isDenied()) {
    if (decision.reason.isPromptInjection()) {
      console.warn("Request blocked due to prompt injection");
      return new Response(
        "Prompt injection detected — please rephrase your message",
        { status: 403 },
      );
    }


    if (decision.reason.isSensitiveInfo()) {
      console.warn("Request blocked due to sensitive information");
      return new Response(
        "Sensitive information detected — please remove it from your prompt",
        { status: 400 },
      );
    }


    return new Response("Forbidden", { status: 403 });
  }


  // Arcjet approved — call your AI provider
  const result = await streamText({
    model: openai("gpt-4o"),
    messages: await convertToModelMessages(messages),
  });


  return result.toUIMessageStreamResponse();
}

Keep denied responses generic – do not leak detector details or explain exactly what was flagged. A generic message asking the user to rephrase is the right default.

Configure prompt injection detection

Section titled “Configure prompt injection detection”

detectPromptInjectionMessage - the text to evaluate. Pass the user’s most recent message, or the full conversation history if you want to scan all messages.

mode - "LIVE" blocks detections. "DRY_RUN" logs detections without blocking. Use "DRY_RUN" to measure the false-positive rate in production before switching to "LIVE". JavaScript detectPromptInjection accepts only mode.

The verdict is binary. In JavaScript, use decision.reason.isPromptInjection() or decision.reason.injectionDetected. In Python, check decision.reason_v2.type == "PROMPT_INJECTION".

Python detect_prompt_injection accepts only mode, which is required. On Python SDK main, omitting mode or passing threshold= raises TypeError.

Combine with abuse protection

Section titled “Combine with abuse protection”

Prompt injection detection controls what your AI model receives. To also block automated clients and enforce per-user budgets, combine it with AI abuse protection and AI budget control.

Prompt injection is one class of AI abuse. Automated traffic is another – if you expose a public AI endpoint, attackers can drive up costs with automated traffic without needing to bypass your system prompt. Bot detection composes cleanly with prompt injection protection:

import arcjet, {
  detectBot,
  detectPromptInjection,
  sensitiveInfo,
  shield,
} from "@arcjet/next";


const aj = arcjet({
  key: process.env.ARCJET_KEY!,
  rules: [
    shield({ mode: "LIVE" }),
    detectBot({ mode: "LIVE", allow: [] }),
    detectPromptInjection({ mode: "LIVE" }),
    sensitiveInfo({
      mode: "LIVE",
      deny: ["CREDIT_CARD_NUMBER", "EMAIL"],
    }),
  ],
});

Protect tool calls

Section titled “Protect tool calls”

The preceding examples protect an HTTP endpoint – the boundary where user input enters your application. But prompt injection can also arrive through tool results: a fetch tool retrieves a page that contains injected instructions, which then re-enter the model context when the tool result is passed back.

Agent guards run the same prompt injection detection inside tool handlers, without a Request object. You can also use a remote policy to apply server-side prompt injection detection to an explicitly mapped input. Install the skill to add an SDK guard automatically:

Terminal window
npx skills add arcjet/skills

Or wire it up manually – create a guard client once at module scope, then call .guard() inline in each tool handler:

import { launchArcjet, detectPromptInjection } from "@arcjet/guard";


const arcjet = launchArcjet({ key: process.env.ARCJET_KEY! });


const piRule = detectPromptInjection();


// Inside your fetch tool handler
export async function fetchTool({ url }: { url: string }, userId: string) {
  const content = await fetch(url).then((r) => r.text());


  const decision = await arcjet.guard({
    label: "tools.fetch",
    metadata: { userId },
    rules: [piRule(content)],
  });


  if (decision.conclusion === "DENY") {
    // Return a safe placeholder rather than the injected content
    return { content: "[Content blocked: prompt injection detected]" };
  }


  return { content };
}

Learn more

Section titled “Learn more”