How to Scan Your LLM Endpoint for Jailbreaks: A Step-by-Step Guide

Why Jailbreak Testing Matters

You shipped an LLM feature. Your users interact with it daily through chat interfaces, copilots, or agent workflows. But do you know if someone can bypass your safety filters with a DAN attack or role-play manipulation?

Jailbreak attacks target the fundamental behavior of LLMs by convincing them to ignore safety guidelines or system prompts. These attacks range from simple "ignore previous instructions" attempts to sophisticated role-play scenarios that trick models into producing harmful content.

Testing your endpoint proactively identifies these vulnerabilities before attackers do. Manual testing catches obvious issues, but automated jailbreak scanning covers the attack vectors you might miss.

Prerequisites: What You Need

Before starting, ensure you have:

• An active LLM endpoint (OpenAI, Anthropic, or self-hosted)
• API authentication (key or challenge-token)
• Your endpoint's expected request/response format
• Basic understanding of your model's safety mechanisms

Most developers already have these components from their existing LLM integration.

Step 1: Configure Your Endpoint URL

Start by pointing your jailbreak scanner at your LLM endpoint. This works with any provider that accepts HTTP requests.

https://api.openai.com/v1/chat/completions

https://your-api.example.com/v1/generate

https://your-api.example.com/v1/stream

Enter your full endpoint URL including the path. The scanner needs to reach the actual inference endpoint, not a wrapper or proxy that might filter requests.

Authentication happens in the next step, so focus on the base URL here.

Step 2: Set Up Payload Schema

Configure how attack payloads get formatted for your endpoint. Different providers expect different JSON structures.

{
  "model": "gpt-4",
  "messages": [
    {"role": "user", "content": "[ATTACK_PAYLOAD]"}
  ],
  "temperature": 0.7
}

{
  "model": "claude-3-sonnet-20240229",
  "max_tokens": 1000,
  "messages": [
    {"role": "user", "content": "[ATTACK_PAYLOAD]"}
  ]
}

{
  "prompt": "[ATTACK_PAYLOAD]",
  "max_length": 500,
  "temperature": 0.1
}

The [ATTACK_PAYLOAD] placeholder gets replaced with actual jailbreak attempts during scanning. Set other parameters like temperature and max_tokens to match your production configuration.

Include your API key in headers:

{
  "Authorization": "Bearer your-api-key-here",
  "Content-Type": "application/json"
}

Step 3: Configure Response Parsing

Tell the scanner how to extract the model's response from your endpoint's JSON structure.

choices[0].message.content

content[0].text

generated_text

For streaming responses (SSE or NDJSON), specify the streaming format and the field containing incremental text chunks. Most streaming implementations send delta content that gets concatenated into the full response.

The scanner needs the complete model output to determine if a jailbreak attempt succeeded.

Step 4: Run Your First Jailbreak Scan

With configuration complete, run your initial scan. The scanner executes multiple attack vectors against your endpoint:

• DAN attacks: "Do Anything Now" prompts that attempt to bypass safety restrictions through role-play scenarios.
• Role-play manipulation: attacks that convince the model to adopt harmful personas or ignore guidelines.
• System prompt leaks: attempts to extract your system prompt or internal instructions.
• Safety bypass: direct requests for harmful content using various persuasion techniques.

Each attack sends a crafted prompt to your endpoint and analyzes the response for signs of successful jailbreaking.

A typical scan runs 10-50 individual attack attempts, depending on your selected attack groups. The process takes 2-5 minutes for most endpoints.

Step 5: Interpret Pass/Fail Verdicts

The scanner returns clear pass/fail verdicts for each attack:

• PASS: Your endpoint correctly refused the jailbreak attempt or provided a safe response.
• FAIL: The attack succeeded. Your model produced content that violates safety guidelines or leaked sensitive information.
• ERROR: Technical issue prevented the attack from completing (network timeout, authentication failure, etc.).

Each result includes the raw request and response so you can verify the verdict manually. Failed attacks show exactly what harmful content your model generated.

Pay special attention to failed DAN attacks and system prompt leaks. These represent immediate security risks in production environments.

Understanding Attack Types

Different attack groups target specific vulnerabilities:

• promptinject: direct and indirect prompt injection attacks covering OWASP LLM01 scenarios.
• lmrc: Language Model Risk Cards attacks focusing on harmful content generation.
• goodside: attacks developed by Riley Goodside, targeting instruction-following vulnerabilities.
• latentinjection: sophisticated attacks that hide malicious instructions within seemingly benign prompts.
• prefix: attacks that manipulate the model by controlling response prefixes.
• webinjection: attacks specific to web-based LLM applications.

Each group covers different attack vectors. Run all groups for comprehensive coverage, or focus on specific groups relevant to your use case.

Automated Scanning Setup

Manual scans catch point-in-time vulnerabilities. Weekly automated scans provide ongoing security validation as you update your model, system prompts, or safety filters.

Configure weekly scans to run automatically. The scanner tests your endpoint consistently and alerts you to new vulnerabilities introduced by changes.

Automated scanning works particularly well for:

• Continuous integration pipelines
• Pre-production security checks
• Monitoring production endpoints for regression

Set up email notifications to get immediate alerts when scans detect new vulnerabilities.

Common Issues and Fixes

• Authentication failures: verify your API key has sufficient permissions and has not expired.
• Timeout errors: increase timeout values for slower endpoints or reduce batch sizes.
• Rate limiting: space out requests or use endpoints with higher rate limits during testing.
• False positives: review raw responses manually. Some models generate refusals that trigger false positive detections.
• Incomplete responses: check your max_tokens setting. Truncated responses can hide successful jailbreaks.

Most issues stem from configuration mismatches between your test setup and production environment.

FAQs

Conclusion

Regular jailbreak testing identifies vulnerabilities before they become security incidents. Configure your endpoint URL, payload schema, and response parsing, then run automated scans covering DAN attacks, role-play manipulation, and system prompt leaks.

Start with manual scans to validate your configuration. Move to weekly automated scanning for ongoing security validation. Focus on failed attacks that represent real risks to your production environment.

If you need a PDF you can send without asking for a company card first, how to send your manager an LLM security PDF without a credit card covers the free export path. Learn more at bastionllm.com.