Red-Teaming an Enterprise AI Agent Before It Reaches Production

An enterprise agent usually reaches its go-live review with an evaluation report attached: accuracy on a held-out set, latency numbers, a sample of reviewed outputs. Those answer whether it does its job. They do not answer the question a CISO is actually asking, which is what it can be made to do by someone who wants it to misbehave.

Those are different exercises with different methods. An evaluation samples the input distribution you expect and reports an average. A red team hunts for one reachable path to an unacceptable outcome and reports it as a finding. An agent can score well on the first and fail badly on the second, because the second is mostly not about the model.

Map the attack surface before you attack it

The OWASP Gen AI Security Project’s GenAI Red Teaming Guide, published in January 2025, frames the work across four areas — model evaluation, implementation testing, infrastructure assessment, and runtime behaviour analysis. That framing is useful precisely because it refuses to let the exercise collapse into prompt tricks against the model.

Start by enumerating, for the specific workflow under test:

• Every tool the agent can call, and what each one does when it succeeds.
• Every permission those tools hold on downstream systems, and whose identity they run under.
• Every data source it reads, including retrieval corpora, and who can write into them.
• Every autonomous action it can take without a human approving it.
• Every boundary it crosses — network egress, other agents, queues, external integrations.

This inventory is itself a finding generator. Teams routinely discover at this stage that a tool built for reading also holds write rights, that a service account is shared across workflows, or that a document library the agent retrieves from accepts uploads from a wider population than anyone assumed.

The classes of attack worth your time

Generic jailbreak prompts are the least interesting part of the exercise for an enterprise workflow. The findings that change designs cluster elsewhere.

Indirect prompt injection. Content the agent retrieves — a document, an email body, a form field, a PDF comment, a filename — carrying instructions aimed at the agent rather than the reader. The test is not whether the model notices; it is whether following the injected instruction can reach anything that matters. That is a permissions question.

Excessive agency. OWASP’s LLM06:2025 Excessive Agency entry names the roots — too many available functions, too many permissions behind them, too much autonomy over high-impact actions — and its remedies double as test cases. Can the agent reach a tool outside its task scope? Does a read tool’s identity also permit writes? Can a high-impact action complete without approval?

Confused-deputy and identity confusion. Can user A’s request cause work to be done under an identity that can see user B’s data? Shared service accounts and cached results are the usual culprits.

Data exfiltration by legitimate means. Not breaking out of the model, but persuading the agent to place sensitive content somewhere a wider audience can read it — a ticket, a shared document, a summary written back to a general-access location, an outbound integration.

Approval-gate erosion. Can a sequence of individually-approved small steps compose into an outcome nobody approved? Can the agent shape the approval request so the reviewer approves something other than what executes?

Denial by resource. On finite local capacity, an input that makes the agent recurse, retry, or generate without bound is a real availability finding, not a curiosity.

Run it under rules, and write the findings down

Before the exercise: agree scope, environment, and rules of engagement in writing. Test against an environment that mirrors production’s permission model — a red team run against a sandbox where every tool is read-only proves very little. Seed the retrieval corpus with the hostile documents you intend to plant, and log them so cleanup is complete.

During: record every attempt, not only the successes. A path that failed tells the next reviewer what has already been covered, and failed attempts are what turn a one-off exercise into a regression suite.

After: write each finding with the reachable outcome, the exact path, the preconditions, and the control that would prevent it. Push fixes toward the layer that can enforce them. A finding fixed by adding a sentence to the system prompt is usually a finding deferred; the same finding fixed by removing a permission, narrowing a tool, or moving authorization into the downstream system is closed. That distinction — model-layer mitigation versus system-layer control — is the single most useful column in the report.

Make it repeatable, because the agent will change

An agent is not a fixed artefact. Its tools grow, its corpus expands, its model gets upgraded, its permissions drift as people solve unrelated problems. A red-team result is therefore a statement about a configuration at a point in time, and it should be recorded as one: which model version, which tool set, which permission model, which retrieval index generation.

Convert the confirmed findings into an automated adversarial suite that runs on every change to any of those, and re-run the human exercise on a schedule and on material change — a new tool, a new data source, a new autonomous action, a new integration. The scheduled repeat is what catches the slow accumulation of capability that no single change request made look risky.

Why private infrastructure changes the exercise

Running the exercise against a self-hosted deployment gives you something a hosted API cannot: the ability to test the infrastructure layer as well as the application. You can inspect what actually leaves the network, verify that a model server is not reachable from where it should not be, examine the logs the platform produced during the attack, and confirm that a stopped workflow really stopped.

It also removes the awkwardness of adversarial testing against someone else’s production service, and keeps the hostile test artefacts — the planted documents, the successful exfiltration payloads, the findings themselves — inside your own boundary, where they belong.

How VDF AI supports pre-production assurance

In a VDF AI Solution deployment, the properties a red team probes are platform configuration rather than model behaviour: which tools a workflow may call, the identity and permission model behind each, where approval gates sit, what the retrieval layer will return to whom, and what the audit record captures. Ask your implementation team to demonstrate a red-team finding end to end — the attempt, the control that stopped it, and the log entry that proves it — before the workflow gets production credentials.