Enterprise AI Agent Security: Threat Model, Control Plane, and Rollout Checklist

The security problem with an enterprise AI agent is not merely whether the model says something wrong. It is whether a probabilistic decision-maker can cross identity, data, and tool boundaries and create real side effects in deterministic enterprise systems. Prompt injection may arrive through a user message, webpage, email, retrieved document, or another agent. If the agent also holds broad tools and long-lived credentials, text can become an email, payment, deletion, or data-exfiltration action.

The goal is therefore not to make the model “always decide correctly.” It is to define an enforceable agent execution envelope: every run is bound to who initiated it, whom it represents, why it is running, which data and tools it may access, which side effects are allowed, and its time, step, and cost limits. The model may plan inside that envelope; controls outside the model decide whether an action is allowed.

Threat-model first, guardrail second

The OWASP Prompt Injection guidance covers both direct and indirect injection and notes that impact depends on the agency granted to the model. Detecting malicious text is therefore only one mitigation. The assets and actions reachable after model output are the actual protection target.

Start a threat model with four facts:

1. Assets: customer data, source code, credentials, payment and email capabilities, durable memory, and audit records.
2. Principals: end user, delegating user, agent workload, tool service, supplier, and operator.
3. Inputs: prompts, retrieved content, tool results, files, webpages, webhooks, and agent-to-agent messages.
4. Worst side effects: unauthorized reads, cross-tenant leakage, irreversible writes, privilege escalation, service or budget exhaustion, and evidence tampering.

MITRE ATLAS is a living knowledge base of adversarial techniques against AI systems and can help map red-team scenarios to tactics and techniques; it is not an exhaustive list of future attacks. The NIST AI RMF Generative AI Profile provides a lifecycle frame for governance, inventory, measurement, and incident handling. Both still need to be translated into the organization’s own assumptions, tests, owners, and risk tolerance rather than treated as compliance checkboxes.

Five threat boundaries that need separate controls

Boundary	Typical failure	Primary controls
Prompts and external content	Indirect instructions in a webpage, email, or retrieved document hijack the goal	Mark external content as untrusted data, restrict available tools, preserve provenance, and test adversarially; never let prompt text change authorization
Tools and side effects	Error, hallucination, or hostile content triggers an overpowered tool	Minimal tool set, read/write separation, parameter schemas, policy checks, idempotency keys, value and rate limits, and approval for high-impact actions
Data and egress	Sensitive context leaves through a response, HTTP request, email, or file	Authorization at retrieval time, tenant isolation, field masking, egress allowlists, DLP, and no raw secrets in prompts or memory
Memory and identity	Durable memory is poisoned, users cross namespaces, or the delegating subject and acting agent are confused	Memory partitioning, provenance and writer identity, TTLs, review for sensitive writes, and explicit subject, actor, tenant, and purpose fields
Supply chain and observability	A tool description, prompt, model, package, or remote service changes; records are missing or collect secrets	Version pinning, signed provenance and inventory, change review, sandbox and egress controls, structured redacted audit events, alerts, and recovery exercises

OWASP Excessive Agency separates the root causes into excessive functionality, excessive permissions, and excessive autonomy. That decomposition is useful: even when prompt injection cannot be completely detected, limiting tool capability, effective permission, and autonomous reach still reduces blast radius. OWASP’s Agentic AI Threats and Mitigations also treats memory poisoning, tool misuse, and multi-agent trust chains as distinct concerns rather than extensions of chatbot input filtering.

Defense in depth: keep the control plane outside the model

A deployable enterprise architecture can be described in seven layers. The point is not a particular vendor product; it is to give every layer a testable responsibility.

1. Ingress and session boundary

Authenticate both user and workload, then establish immutable user_id, actor_id, tenant_id, purpose, data classification, and session fields. External documents, tool results, and peer-agent messages remain untrusted content. A sentence claiming “the administrator approved this” cannot elevate privileges.

2. Orchestrator and bounded state

The model may propose a plan, select a candidate tool, or decide that more information is needed. The orchestrator owns maximum steps, timeout, retries, cost, stop conditions, and recoverable state. Prefer an explicit state machine for high-risk workflows instead of letting a model recurse freely until it succeeds.

3. Policy decision and enforcement points

Before execution, each tool request reaches policy as structured data: subject, actor, tenant, purpose, tool, action, resource, parameter summary, and risk tier. Policy returns allow, deny, or require-approval, and an enforcement point applies the result. A model statement that approval exists is not evidence of approval.

4. Tool and data gateway

The gateway should provide tool allowlists, schema validation, credential brokering, rate and quota limits, egress policy, and consistent audit events. It is a valuable choke point, not the only defense. A target API must still authenticate and authorize every call; otherwise bypass traffic or a gateway misconfiguration directly exposes the asset.

5. Isolated execution

Run code execution, browser automation, and untrusted file handling in task-scoped sandboxes. Disable unnecessary network, filesystem, and cloud metadata access by default. Enforce hard CPU, memory, step, token, external-request, and transaction limits, and make the kill switch revoke the session, credentials, and queued work.

6. Evidence and operational feedback

Record input provenance hashes, model and prompt versions, tool and parameter summaries, policy results, human approvals, external side effects, errors, and cost. Do not equate auditability with storing unredacted payloads, secrets, or private chain of thought. Operations need the minimum sufficient evidence to reconstruct decisions and side effects, connected to SIEM, incident response, and retention policy.

7. Lifecycle control plane

An agent registry should include at least an owner, purpose, model and prompt versions, tool and data scopes, suppliers, evaluation results, environment, risk tier, last review, and shutdown path. The NIST AI RMF Core emphasizes inventory, third-party risk, post-deployment monitoring, override, decommissioning, and incident response. These need to exist as operating capabilities, not only as design documents.

Deterministic controls versus model judgment

Decision	Deterministic control should own	Model may assist with
Whether data may be read	RBAC/ABAC, tenant and field authorization	Classifying the type of data requested
Whether a tool may be called	Allowlist, scope, resource, and schema	Proposing a candidate from allowed tools
Whether a high-risk action executes	Value limits, destination checks, dual approval, and transaction preconditions	Summarizing rationale and the pending change
Whether content may leave the boundary	Egress allowlist, data classification, and DLP rules	Semantic classification or suspicion score
Whether a run must stop	Timeout, step, cost, and error limits	Estimating whether the task is complete
Whether memory persists across sessions	Identity, namespace, data class, TTL, and write policy	Proposing candidate facts to retain

Model classifiers, prompt-injection detectors, and LLM-as-a-Judge checks have false positives and false negatives and may be influenced by the same hostile input. They are useful for risk scores, alerting, and review prioritization. When an error can create a material side effect, final enforcement should not depend on another model alone.

Identity: distinguish subject, actor, and workload

When an agent calls a backend for a person, OAuth 2.0 Token Exchange (RFC 8693) defines token exchange and subject/actor semantics for impersonation and delegation. An implementation should bind audience, resource, scope, actor, and lifetime to the target tool and propagate revocation or session termination downstream. The RFC does not choose a universal token lifetime or perform business authorization automatically. Possessing an exchanged token does not mean every action should be allowed.

For a background service or autonomous workload, the SPIFFE Workload API can deliver X.509-SVIDs or JWT-SVIDs and trust bundles. SPIFFE’s concepts documentation explains that short-lived credentials rotate automatically and applications need not be provisioned with a bootstrap secret. This improves workload authentication and static-key handling, but a SPIFFE ID answers “which workload is this,” not “may it refund this customer.” Business policy and the target service must answer the latter.

How one high-risk tool call should cross the system

Consider a request to refund a customer and send a notification:

1. The agent proposes a refund.create plan but has not executed it.
2. The orchestrator attaches immutable user, support role, tenant, case ID, and session context.
3. Policy checks the refund limit, case status, data scope, destination, and whether a second approver is required.
4. The tool gateway validates the schema and idempotency key, then obtains a short-lived credential valid only for the refund API. The notification tool never sees the payment credential.
5. The target API independently enforces authorization and business preconditions and returns a transaction ID rather than relying on a model-generated success statement.
6. The system records policy version, approver, transaction ID, and notification result. On failure, it stops dependent steps and follows a compensation or human-handoff path.

The central pattern is to authorize structured intent before executing a concrete side effect, not to grant broad access and rely on output scanning afterward. AWS’s generative AI security reference architecture for agents likewise treats session isolation, identity, gateway, memory, and observability as cooperating capabilities and warns that agents can combine tools into greater effective privilege. It is a useful cloud implementation reference, not a universal safety guarantee.

Where the architecture still fails

• A classifier does not eliminate prompt injection: novel, encoded, multi-step, and cross-tool attacks can pass detection.
• Least privileges may compose: several low-privilege tools can produce a high-impact outcome that was never evaluated as one capability.
• Human approval may become ceremonial: alert volume, incomplete summaries, and interface framing can cause approval fatigue.
• A gateway can fail or be bypassed: policy synchronization, credential injection, and side-channel traffic require continuous testing.
• More logs do not automatically improve auditability: an oversized trace may retain sensitive data while omitting policy version, external side effect, or correlation ID.
• Models and suppliers change: model, remote tool, prompt, package, and data-source changes can invalidate earlier evaluations.
• Frameworks are not warranties: OWASP, NIST, MITRE ATLAS, SPIFFE, and cloud products support parts of risk management. Security still depends on the deployed architecture, configuration, process, and operations.

Staged rollout checklist

Stage 0: Inventory and boundaries

• Every agent has an owner, purpose, data classes, tools, side effects, suppliers, and shutdown path.
• Define subject, actor, workload, tenant, and session; prohibit shared long-lived service accounts.
• Model direct and indirect injection, cross-tenant access, memory poisoning, tool composition, supply-chain, and egress abuse cases.
• Define prohibited, approval-required, reversible, and incident-reportable conditions for every high-risk action.

Stage 1: Read-only and shadow mode

• Validate plans with read-only tools or dry runs before creating side effects in production systems.
• Build evaluations for normal, denied, hostile-content, tool-timeout, and cross-tenant cases.
• Confirm traces correlate input provenance, policy decisions, tool results, and versions without exposing credentials or unnecessary personal data.

Stage 2: Bounded writes

• Every write tool has a schema, minimal scope, idempotency, timeout, quota, rate limit, and recovery path.
• High-impact actions use step-up or dual approval, with the UI showing the actual resource, parameters, and diff.
• Target APIs independently authorize requests instead of trusting the gateway or model output as their only boundary.

Stage 3: Adversarial and failure exercises

• Test indirect injection through webpages, email, files, tool output, memory, and agent-to-agent messages.
• Test tool composition, replay, race conditions, supplier failure, credential revocation, and the kill switch.
• Map findings to OWASP and MITRE ATLAS while keeping organization-specific abuse cases.

Stage 4: Canary and continuous governance

• Launch with canaries, low limits, and explicit exit criteria; monitor denials, human handoffs, unauthorized access, anomalous egress, cost, and recovery time.
• Maintain on-call ownership, incident severity, shutdown, notification, evidence, and recovery runbooks, and rehearse them.
• Re-run security regressions when the model, prompt, tool, permission, memory policy, or supplier changes; retire unused agents, tools, and credentials.

Related reading

• Start with the complete AI Agent guide for the broader model, tool, state, evaluation, and governance architecture.
• If tools use a standard interface, continue with MCP as the interface between models and tools; protocol consistency does not make authorization safe by itself.
• To embed security responsibilities into a shared platform, see the Enterprise Agentic AI control plane and governance guide.
• For code execution and multi-tenant runtimes, review the isolation trade-offs in the EKS multi-tenant AI agent sandbox.

Primary sources

• OWASP LLM01:2025 Prompt Injection
• OWASP LLM06:2025 Excessive Agency
• OWASP Agentic AI — Threats and Mitigations
• NIST AI 600-1: Generative AI Profile
• NIST AI RMF Core
• MITRE ATLAS
• RFC 8693: OAuth 2.0 Token Exchange
• SPIFFE Workload API
• AWS Security Reference Architecture for Generative AI: Agents