How LLM Firewalls Work: Scanning Prompts, Verifying Outputs, and Firewalling AI Agents

Most teams secure their AI by picking a well-aligned model and writing a careful system prompt. That helps, but it protects only one layer. The model is still reachable by anyone who can send it text, and once you connect it to tools and let it take actions, a single crafted message can do real damage. An LLM firewall closes that gap by treating model safety as a runtime security problem, not only a training problem.

This post explains the architecture in plain terms: what an LLM firewall is, what it catches, the four checkpoints it inspects, how it stays fast, and why the firewall itself should be deterministic rather than another AI agent.

What is an LLM firewall?

An LLM firewall is a security layer that inspects the traffic flowing into and out of a language model and returns a structured verdict that decides whether the request proceeds. It is the same idea as a network firewall or a web application firewall, applied to natural-language traffic instead of packets or HTTP requests.

In practice it is a service that sits in front of your model. Your application calls the firewall, the firewall returns a verdict such as ALLOW, FLAG, or BLOCK, and your code uses that verdict to decide what happens next. Because it inspects text rather than model internals, the same firewall can sit in front of any model from any provider.

Why isn't a safe model enough?

A well-aligned model reduces the odds of a harmful response, but it does not eliminate them, and it does nothing about the parts of the system the model never controlled. Three gaps remain at runtime.

• Models can be talked off-policy. Prompt injection and jailbreaks are an ongoing arms race. New phrasings, role-play framings, and encodings keep working against models that refuse the obvious version of the same request.
• A safe response can still be a leak. Whether a response is dangerous often depends on deployment context, not the words alone. The model has no idea what is in your system prompt, what data it can reach, or what the surrounding application will do with its output.
• Tools and actions are outside the model's safety training. Once the model can call APIs, write files, or run commands, the risk is no longer “did it say something bad,” it is “did it do something bad.” Alignment training was never about your file system.

The reframing that makes a firewall useful is simple: alignment is a property of the model, but security is a property of the deployed system. You need a control that does not depend on the model behaving correctly, because the whole point is to catch the cases where it does not.

What threats does runtime scanning catch?

Runtime scanners group adversarial inputs into a handful of threat classes. Splitting them up matters because each class has a different signature, and a single specialized detector per class beats one generic classifier trying to do everything.

The last class is the newest and the most forward-looking. As models gain autonomy, treating alignment as something to watch for at runtime, not just to train for, becomes a practical security concern rather than a research abstraction.

How does an LLM firewall work? The four checkpoints

A complete firewall inspects four points in the request lifecycle. The first two protect a plain chat application. The last two matter the moment the model can use tools or take actions.

1. Scan the input

Before any prompt reaches the model, the firewall scans it across the threat classes above and returns a verdict. ALLOW passes through, FLAG proceeds but is logged and may trigger step-up handling, and BLOCK stops the request before it ever touches the model.

POST /scan/input

{
  "verdict": "BLOCK",
  "category": "exfiltration",
  "reason": "System-prompt extraction via role-play framing"
}

2. Call the model

If the input is allowed, your application forwards the prompt to whatever model you use. The firewall is vendor-neutral, so this step is unchanged from a normal integration. One firewall integration can front every model in your stack.

3. Verify the output

Before the response reaches the user, the firewall checks it for signs that the model was successfully manipulated during generation: leaked secrets, content that indicates a jailbreak landed, or evidence of alignment subversion that slipped past the input scan. The verdict here is closer to CLEAN or COMPROMISED. Output verification is what catches the attacks that the input scan missed, which is why it matters even though many products skip it.

4. Inspect tools and MCP content

When a model connects to external tools through a protocol like MCP, the tool descriptions and tool responses are read as text and enter the model's context window. That makes them an injection vector: a poisoned tool can hide instructions in its description or its output to hijack the agent. The firewall inspects this content before it reaches the model and flags tool poisoning, indirect injection, and hidden exfiltration payloads.

POST /scan/tool

{
  "verdict": "POISONED",
  "threat": "tool_poisoning",
  "reason": "Hidden exfiltration instruction in tool description"
}

The agent action firewall

For agents that take actions, there is a fifth control worth calling out on its own. Before any side-effecting action executes (a file write, an API call, a shell command), the firewall intercepts it and classifies it by blast radius and reversibility. Destructive, irreversible actions are blocked automatically. Reversible ones can queue for human approval. This is the difference between “the agent suggested something bad” and “the agent did something bad,” and it is the layer that turns an autonomous agent into something you can actually run in production.

How does it stay fast enough for production?

A firewall on the critical path has to be fast, or nobody ships it. The standard answer is a staged pipeline that spends cheap work first and expensive work only when it is warranted.

1. Normalize. Decode and canonicalize the input so obfuscation tricks (unusual encodings, whitespace, homoglyphs) cannot slip past pattern matching.
2. Deterministic pre-screen. A fast regex and heuristic pass catches the obvious attacks in microseconds to milliseconds and can short-circuit straight to a block without spending any model calls.
3. Parallel detectors. When the pre-screen is not conclusive, the specialized detectors (one per threat class) run at the same time rather than in sequence, so total latency is the slowest detector, not the sum.
4. Fixed aggregation. A fixed rule combines the individual findings into one verdict. The most severe finding wins; there is no model deciding the final answer.

The result is a single structured verdict, typically in well under a few seconds, with the expensive detectors skipped entirely on the easy cases. The latency cost is real, so teams usually scale how much scanning they apply to the blast radius of what is being protected: light on a read-only chat, heavy in front of a tool that can delete data.

Why should the firewall be deterministic, not an autonomous agent?

It is tempting to build the security layer out of an LLM too: “ask a smart model to judge whether this request is safe.” That is the wrong shape for a security control, and the recommended pattern is the opposite: deterministic middleware with a fixed pipeline, not an agent that autonomously decides what to do.

There are three reasons this matters.

• Predictability. The same input should always produce the same verdict. A bounded pipeline with fixed aggregation gives you that. An autonomous judge does not.
• Auditability. When something gets through, you need to reconstruct exactly why the verdict was what it was. Fixed logic is reviewable. “The model felt it was fine” is not.
• No new attack surface. An LLM in the security path is itself susceptible to prompt injection. You would be guarding the model with another model that has the same weakness. The detectors can use models internally, but the control flow around them must be fixed code.

The phrase to keep in mind is structured security middleware. Models can do the judging inside individual bounded detectors, but they should never be the thing that decides what action to take on your behalf.

How do you integrate one?

Integration is a gateway pattern: wrap your existing model call with a scan before and a scan after, and branch on the verdict. The control flow is the whole point, and it stays in your code where you can reason about it.

const input = await firewall.scanInput(userPrompt)
if (input.verdict === "BLOCK") return reject(input.reason)
if (input.verdict === "FLAG") logAndStepUp(input)

const answer = await llm.generate(userPrompt)   // any model

const output = await firewall.scanOutput(answer)
if (output.verdict === "COMPROMISED") return reject()

return answer

For an agent, you add the same pattern around tool content and around each action: scan a tool response before it enters the context window, and classify an action before you execute it. The model stays vendor-neutral throughout, so swapping models or running several in parallel does not change the security code.

What are the limits?

A firewall is a strong layer, not a silver bullet. Worth being honest about the tradeoffs.

• False positives versus recall. Tighten detection and you block more attacks but also more legitimate requests. The threshold is a product decision, not a fixed setting, and it differs by how costly a miss would be.
• Latency is a real cost. Even a fast pipeline adds time and compute. The staged design keeps it small, but it is never free, which is why scanning intensity should track the blast radius.
• It is one layer of defense-in-depth. A firewall complements model safety training, least-privilege tool design, scoped credentials, and human approval gates. It does not replace any of them.
• The threats evolve. New injection techniques appear constantly, so detectors need ongoing updates. A firewall buys you a place to deploy those updates centrally instead of re-prompting every application.

Frequently asked questions