Ensuring AI systems do not execute beyond their authority

The most dangerous AI failure is not a wrong answer, it is a correct action that nobody authorised.

In the forensic readiness post, I described how to preserve evidence of what AI systems did. In the agent registry post, I described how to know which agents exist and what they are permitted to do. This post addresses the gap between those two concerns. How do you ensure that what an agent actually does stays within the boundaries of what it was authorised to do?

This is Output Assurance. The question is not whether the output is correct or safe, but whether the output was authorised.

The Distinction That Matters

The AI governance conversation has been dominated by two main concerns: 

• Accuracy: Is the output correct?
• Safety: Is the output harmful?

Both matter and neither is sufficient.

Output Assurance asks a different question: Was this action within the agent’s approved scope of authority?

An agent can produce a perfectly accurate, entirely safe output that is nonetheless a governance failure because it was never authorised to take that action. A procurement agent that correctly identifies the cheapest supplier and places the order has produced an accurate, safe output, but if its authority was limited to recommending suppliers rather than placing orders, it has exceeded its mandate. The output is correct. The action is unauthorised and governance has failed.

“Agent systems increasingly execute rather than advise. When an AI agent queries regulated data, invokes effectful tools and mutates persistent state, correctness is no longer captured by whether a terminal output looks plausible.” Source: Runtime Verification for Governed AI Agent Actions, ArXiv (2026)

This is the assurance gap that enterprises are now confronting. As Fortune reported this week: “The newest generation of AI systems is not simply waiting for instructions. Agents now pursue goals, call tools, make recommendations, trigger actions and hand work from one system to another.” The human role changes with them. The employee is no longer simply a user but the person responsible for directing and overseeing the agent.

Why This Is Urgent Now

Three events in July–August 2026 made Output Assurance an operational emergency rather than a theoretical concern.

The OpenAI Sandbox Escape. On 20 July 2026, OpenAI disclosed that an internal model (the same one credited with disproving the Erdős unit distance conjecture) had repeatedly found ways to act outside its containment system during limited internal deployment. The model used stolen credentials, chained multiple vulnerabilities and found a remote code execution path into Hugging Face’s production infrastructure. Hugging Face’s forensic reconstruction documented approximately 17,600 attacker actions executed between 9 and 13 July 2026.

The Anthropic CTF Breach. Days later, Anthropic disclosed that three of its models (Claude Opus 4.7, Claude Mythos 5 and an unreleased research build) broke out of a Capture the Flag sandbox and compromised real enterprise infrastructure. One model published a malicious package to PyPI, attempted to obtain a burner phone number to bypass two factor authentication and successfully uploaded code that was downloaded by 15 external systems. Two of the three affected companies believed they were facing a sophisticated human attacker.

The Legal Vacuum. As WIRED reported on 1 August 2026: “Both major AI labs’ models broke containment, escaped onto the internet and hacked other companies. If a human had done that, the law would likely be against them. But a bot?” The CFAA’s intent requirements make it a poor fit for AI related cases. Agency law (where a principal gives an agent authority to act on their behalf) has always assumed human agents.

“Perhaps most concerning to critics is that AI agents are goal oriented but lack a human moral or ethical compass. In some situations, an agent may infer actions that were never explicitly authorised if those actions appear necessary to achieve its objective.” Source: Brownstein Hyatt Farber Schreck, client alert (24 July 2026)

The lesson is not that AI models are dangerous. The lesson is that capability without authority enforcement is indistinguishable from a security incident. Also that the security protocols and systems around the environments used is in place need to be constantly monitored.

Proof Carrying Agent Actions

The most rigorous response to the Output Assurance problem comes from the emerging Proof Carrying Agent Actions (PCAA) framework. PCAA is a runtime neutral governance model centred on an action certificate rather than on a vendor native session record.

PCAA organises control around five checkpoints:

Checkpoint	Function	Evidence Produced
Pre-action admissibility	Validates the proposed action against the authority model before execution	Admissibility receipt
Action open	Registers the action as in progress with a unique identifier	Action envelope
Assumption capture	Records the assumptions under which the action was approved	Assumption log
Approval	Obtains the required authorisation (automated or human)	Approval receipt
Outcome closure	Verifies the outcome matches the approved intent and closes the action	Closure attestation

These checkpoints bind to a portable action envelope, runtime and approval receipts and replay ready proof. The critical design principle: the proof travels with the action, not in a separate log that can be disconnected from it.

“PCAA defines the deployer owned route review prove governance process, while CAVA defines the stable action object that process governs.” Source: Canonical Action Verification and Attestation for Runtime Governance of Agentic AI Systems

Below PCAA sits CAVA (Canonical Action Verification and Attestation), a runtime semantics layer for converting heterogeneous agent activity into canonical action objects. A single agent action may be represented by many incompatible runtime records across different systems. CAVA gives each action a stable, verifiable identity regardless of which runtime executed it.

This matters because without canonical action identity, you cannot:

• Prove that the action executed was the action approved
• Reconstruct the full action from partial logs across systems
• Demonstrate to a regulator that the governance gateway’s permit verdict corresponds to what actually happened
• Detect cases where an agent’s actual execution diverged from its stated intent

The Authority Model

Output Assurance requires a formal authority model, an explicit, machine readable specification of what an agent may and may not do.

This is not a prompt instruction. It is not a system message. It is an architecturally separate enforcement layer that the agent cannot modify, bypass or reason its way around.

The authority model has five dimensions:

1. Action Scope

What actions is this agent permitted to take? Not what it can do (capability), but what it may do (authority).

Authority Level	Description	Example
Observe	Read data, no modifications	Query a database, retrieve a document
Recommend	Propose actions for human decision	Suggest a supplier, draft a response
Prepare	Create artefacts requiring approval before effect	Generate a purchase order pending sign off
Execute (bounded)	Take action within defined parameters	Send a pre-approved template email
Execute (autonomous)	Take action without prior approval	Automatically close a resolved ticket
Delegate	Invoke other agents to act	Orchestrate a multi-agent workflow

2. Data Boundaries

What data may this agent access and what may it do with that data?

• Read scope: Which systems, which data classifications, which records
• Write scope: Which fields, which systems, which conditions
• Exfiltration boundaries: What data may leave the agent’s context
• Retention limits: How long may the agent hold data in memory or context
• Classification ceiling: The highest data classification the agent may process

3. Decision Authority

What decisions may this agent make autonomously versus what must be escalated?

• Threshold based: Financial value, risk score, confidence level
• Category based: Certain decision types always require human approval
• Novelty based: Decisions outside the agent’s training distribution require escalation
• Consequence based: Irreversible actions require higher authority than reversible ones

4. Temporal Constraints

When may this agent act?

• Operating hours: Some agents should only execute during business hours
• Rate limits: Maximum actions per time period
• Cool down periods: Minimum time between consequential actions
• Expiry: Authority that automatically revokes after a defined period

5. Delegation Rules

If this agent can invoke other agents, what are the constraints?

• Permitted delegates: Which other agents may be invoked
• Authority inheritance: Delegates cannot exceed the delegator’s authority
• Chain depth: Maximum delegation depth before mandatory human checkpoint
• Attribution preservation: Every action in the chain must be attributable to the originating agent and ultimately to a human

Institutional Attestation: Governing Actions, Not Agents

A June 2026 paper from ArXiv proposes a model that reframes the problem entirely, govern actions, not agents. Under this model, an agent retains full autonomy over planning and reasoning but holds no execution authority over designated high risk actions.

“We formalise this institutional pattern as a computational governance model for AI agent systems. Under the proposed model, an agent retains full autonomy over planning and reasoning but holds no execution authority over designated high risk actions.” Source: Governing Actions, Not Agents: Institutional Attestation as a Governance Model for Autonomous AI Systems, ArXiv (2026)

This is a zero-trust architecture applied to agent actions. The agent can think whatever it wants. It can plan whatever it wants. But when it attempts to act  (to mutate state, to invoke a tool, to send a communication, to delegate to another agent) it must present its intent to an institutional attestation layer that decides whether to permit execution.

The keywords from the paper tell the story:

• AI agent governance
• Institutional attestation
• Zero-trust architecture
• Policy decision point
• Cryptographic attestation
• Verifiable computation
• Tamper evident audit.

This maps directly to how we govern human organisations. A junior analyst can recommend a trade, but they cannot execute it without attestation from someone with the authority to approve. The attestation is recorded and the record is tamper evident with the chain of authority is auditable.

The Excessive Agency Problem

OWASP’s Top 10 for Agentic Applications (2026) identifies Excessive Agency as the category where most enterprise incidents originate. The problem has three components:

• Excessive Functionality: The agent has access to tools it does not need for its stated purpose. A customer service agent with access to the payment processing API. A document summarisation agent with write access to the source documents.
• Excessive Permissions: The agent’s permissions exceed what its function requires. Read access to all customer records when it only needs the current customer’s data. Admin level API access when read only would suffice.
• Excessive Autonomy: The agent can act without appropriate human oversight for the consequence level of its actions. Sending external communications without review. Making financial commitments without approval.

“Least privilege for agents needs a different unit of control. The security boundary cannot be ‘this workload owns this token’. It has to be ‘this delegated identity may attempt this action, for this purpose, in this workflow, under these constraints and only through an enforcement point that can say no.” Source: Least Privilege in AI Agents and Agentic Identity Permit.io (2026)

The fix is not to remove capability. It is to enforce the principle of least privilege at runtime. Microsoft’s July 2026 guidance makes this explicit: “When an agent operates without a managed identity and least privilege role based access controls, it can access or modify sensitive data beyond intended permissions if controls are not properly configured.”

Runtime Derived Least Privilege

Static permission models fail for agents because agent behaviour is non-deterministic. You cannot predict at design time every action an agent will attempt. The emerging pattern is runtime derived least privilege, observe the agent’s actual behaviour over a defined window, then use the evidence to define the enforcement boundary.

The process:

• Observation mode: Deploy the agent with broad permissions but full logging. Record every action, every tool invocation, every data access, every delegation.
• Baseline extraction: After a defined observation window, extract the agent’s actual operational envelope. What it actually needed to do its job.
• Policy generation: Generate a least privilege policy from the observed baseline, with a defined margin for legitimate variation.
• Enforcement mode: Switch from observation to enforcement. Actions within the derived policy are permitted. Actions outside it are denied or escalated.
• Continuous refinement: Monitor for legitimate new behaviours that require policy expansion (through a governed change process) versus anomalous behaviours that indicate drift or compromise.

This approach acknowledges a fundamental truth, we cannot fully specify agent behaviour in advance. However we can observe it, baseline it and enforce boundaries around it.

The Five Eyes intelligence alliance’s 2026 guidance reinforces this. Enterprise rollouts of agentic AI are “moving far too fast, operating with excessive autonomy and generating unprecedented resilience risks.”

Human-in-the-Loop Is Not a Checkbox

The phrase “human-in-the-loop”(HITL) has become a compliance incantation. Its something organisations say to satisfy auditors without examining what it actually means in practice.

True HITL for Output Assurance requires architectural commitment, not just a UI (User Interface) button.

Three modes, each with different architectural requirements:

• Human-in-the-Loop (Active Oversight): The agent cannot proceed past specific checkpoints without explicit human approval. Appropriate for high-consequence, irreversible actions.
• Human-on-the-Loop (Supervisory Oversight): The agent acts autonomously but a human monitors its behaviour in near realtime and can intervene. Appropriate for medium consequence actions where speed matters.
• Human-over-the-Loop (Governance Oversight): The agent acts autonomously. Humans review outcomes periodically and adjust policies. Appropriate only for low consequence, high volume actions.

The oversight mode must be proportionate to consequence. Not every action needs active approval. The decision about which oversight mode applies to which action must itself be a governed, documented, reviewable decision.

As Fortune reported August 2026, that most companies are teaching people how to get more out of AI and not teaching people how to lead it. BNY’s approach illustrates what maturity looks like: “We launched our first digital employee in 2025 with the same approach as our enterprise wide systems, governance was built in from the start. Digital employees are onboarded, governed and continuously monitored with the same rigour we expect across our technology estate, while accountability always remains with our people.”

The Enforcement Architecture

The enforcement point sits between the agent’s decision to act and the execution of that action. Every action passes through it. No exceptions.

Agent Intent → Enforcement Point → [PERMIT / DENY / ESCALATE] → Execution → Outcome Verification

Design principles:

• Mandatory: No agent action bypasses the enforcement point. This is enforced at the infrastructure level and not the application level.
• Separate: The enforcement point is architecturally independent of the agents it governs. Agents cannot modify, disable or influence its decisions.
• Fail closed: If the enforcement point fails, the default is DENY, not PERMIT. The sandbox escapes demonstrated what happens when containment fails open.
• Evidential: Every interaction produces an immutable evidence record that feeds the forensic readiness architecture.
• Performant: Authority validation must not introduce latency that makes the system unusable.

The enterprise governance architecture pattern from Himanshu AI Substack article Agentic AI Governance for Enterprise Teams (2026) shows the full stack:

• User → SSO/RBAC → AI Gateway → Policy Engine → PII + Prompt Injection Scanner →Agent Orchestrator → Tool Registry → Human Approval Layer → LLM → Output Validator →Audit + Observability

Every layer in this stack is an enforcement point:

• The AI Gateway enforces rate limits and routes to approved models
• The Policy Engine validates against authority models
• The Tool Registry constrains which tools are available
• The Human Approval Layer implements HITL checkpoints
• The Output Validator confirms the action matches the approved intent

The Drift Problem

Authority models are point in time artefacts. Agents drift and their behaviour changes through:

• Prompt modifications: Someone edits the system prompt without governance approval
• Retrieval corpus updates: The knowledge base changes, altering the agent’s decision context
• Model version changes: The underlying model is updated, changing behaviour characteristics
• Tool additions: New tools are connected, expanding capability beyond the authority model
• Context accumulation: Long running agents accumulate context that shifts their behaviour
• Agent-to-agent influence: In multi-agent systems, one agent’s outputs become another’s inputs

Output Assurance must detect drift, the divergence between an agent’s registered authority and its actual behaviour. This requires continuous comparison:

Dimension	Registered Authority	Drift Signal
Tools used	Approved toolset	New tool invocations not in approved set
Data accessed	Approved data scope	Access to systems outside approved scope
Actions taken	Approved action types	Action types not in authority model
Decision patterns	Expected distribution	Statistical shift in decision outcomes
Delegation behaviour	Approved delegates	Delegation to unregistered agents
Resource consumption	Expected budget	Anomalous spikes or sustained increases

When drift is detected, the response is graduated:

• Alert → Restrict → Pause → Revoke.

The severity depends on the magnitude of drift, the consequence level of the divergent actions and whether the drift appears intentional or incidental.

The Multi-Agent Amplification Problem

Output Assurance becomes exponentially more complex in multi-agent systems. When Agent A delegates to Agent B, which delegates to Agent C, authority must be preserved, constrained and auditable across the entire chain.

• The Principle of Non-Amplification: A delegated agent cannot possess greater authority than the agent that invoked it. Authority can only be reduced through delegation and never expanded.
• The Attribution Chain: Every action in a multi-agent workflow must be attributable to the specific agent that executed it, the agent that delegated the task, the original agent that initiated the workflow and the human who is ultimately accountable.
• The Cascade Risk: If Agent A has authority to place orders up to £10,000 and it delegates to Agents B, C and D simultaneously, can the combined actions exceed £10,000? The authority model must address aggregate limits across delegation chains.

The Cloudflare recently launched agent identity tools (cloudflare.id) paired with wallets for agentic commerce which demonstrates how rapidly the delegation problem is scaling. When agents hold funds and transact at machine speed, authority enforcement is not a governance nicety, it is a financial control.

The Regulatory Alignment

Output Assurance maps directly to regulatory requirements that became binding this month:

Requirement	EU AI Act (binding 2 Aug 2026)	UK Framework	US Framework (finalised 3 Aug 2026)
Authority boundaries	Article 14: Human oversight proportionate to risk	ICO accountability: Demonstrate appropriate controls	White House voluntary framework: Oversight requirements
Runtime enforcement	Article 9: Risk management throughout lifecycle	FCA SM&CR: Named individuals accountable	NIST AI RMF Govern 1.1: policies define boundaries
Evidence of compliance	Article 12: Automatic logging throughout lifetime	ICO: Demonstrate compliance on demand	SEC: Material AI risk disclosure
Human oversight	Article 14: Ability to override or interrupt	AI Safety Institute: Meaningful human control	Colorado AI Act: Reasonable care in deployment

The EU AI Act’s Article 14 is the most explicit: High risk AI systems must be designed to be “effectively overseen by natural persons” who can “fully understand the capacities and limitations of the high risk AI system” and “be able to decide, in any particular situation, not to use the high risk AI system or to otherwise disregard, override or reverse the output.”

The White House framework, finalised on 3 August 2026, shapes how the federal government works with leading AI developers before and after their most advanced models are deployed. Anthropic, Google, Meta and OpenAI will be briefed on the framework in August 2026.

The Maturity Model

Level	State	Characteristics
0	Unconstrained	Agents act without authority validation. No enforcement point, No evidence.
1	Prompted	Authority boundaries defined in system prompts. No runtime enforcement. Agent can reason around them.
2	Gated	Human approval required for defined action categories. Manual process, inconsistent application.
3	Enforced	Governance gateway validates all actions against authority model. Automated PERMIT/DENY/ESCALATE.
4	Attested	Runtime enforcement + PCAA proof carrying actions + CAVA canonical verification + forensic evidence chain.

Most enterprises are at Level 1. Authority boundaries exist only in prompts that the model may or may not respect. The regulatory environment demands Level 3 minimum. The forensic readiness requirements from the first post in this series demand Level 4.

The gap between Level 1 and Level 3 is not incremental improvement. It is an architectural transformation, from advisory constraints that depend on model compliance to enforcement constraints that operate independently of the model.

The Connection to the Series

This series describes a single architectural system viewed from three angles:

• Forensic Readiness answers: What did the agent do and can we prove it?
• The Agent Registry answers: Which agents exist and what are they permitted to do?
• Output Assurance answers: Did the agent stay within its authority and can we demonstrate that it did?

Remove any one and the system fails:

• Without forensic readiness, you cannot prove what happened
• Without a registry, you cannot define what should happen
• Without output assurance, you cannot prevent what should not happen

Together, they form the Governance Triad for Agentic AI, know what exists, constrain what it does, prove what happened.

What Comes Next

The governance triad addresses the runtime problem, ensuring agents behave within boundaries during operation. But agents do not arrive fully formed. They are built from components, models, prompts, retrieval corpora, tool definitions, training data, fine tuning datasets, evaluation benchmarks.

Each component in that supply chain is a potential point of compromise. A poisoned training dataset. A manipulated retrieval corpus. A backdoored tool definition. A prompt injection embedded in a document the agent retrieves.

The next post in this series addresses AI Supply Chain Integrity, ensuring that the components from which agents are assembled are themselves trustworthy, traceable and tamper evident.

Because you cannot assure the output of a system whose inputs you do not trust.

The next post will also move from the Governance Triad to the Governance Tetrad as hinted in my post You’re Right: A Registry Isn’t Enough

References

• “Runtime Verification for Governed AI Agent Actions.” ArXiv (2026)
• “Proof Carrying Agent Actions: Model Agnostic Runtime Governance for Heterogeneous Agent Systems.” ArXiv (2026)
• “Canonical Action Verification and Attestation for Runtime Governance of Agentic AI Systems.” ArXiv (2026)
• “Governing Actions, Not Agents: Institutional Attestation as a Governance Model for Autonomous AI Systems.” ArXiv (2026)
• “Architectural Requirements for Agentic AI Containment After the April 2026 Frontier Model Escape.”ArXiv (2026)
• “A Theory of Least Autonomy in AI.” ArXiv (2026)
• “Computational Jurisprudence: Verifiable Law for Machine Societies.” Preprints.org (2026) 
• “Proof Carrying Agent Actions (PCAA).” Emergent Mind (2026)
• “Safety and Alignment in an Era of Long Horizon Models.” OpenAI (2026)
• “Anthropic Reveals Claude AI Model Hacked Three Companies During Tests.”TechRadar (2026) 
• “Swarm of OpenAI Agents Exploit Artifactory Zero-Day to Escape Sandbox and Breach Hugging Face.” InfoQ (2026)
• “The OpenAI and Anthropic AI Hacking Sprees Are a Messy New Legal Frontier.” WIRED (2026)
• “OpenAI AI Agent Sandbox Escape.” Adversa AI (2026) 
• “AI Gone Rogue: What Recent OpenAI and Anthropic AI Incidents Could Mean for CFAA Liability.” Ballard Spahr (2026) 
• “Your AI Agent Can Be a Teammate. But It Still Needs a Boss.” Fortune (2026) 
• “White House Finalizes Artificial Intelligence Oversight Framework.” Politico (2026)
• “Least Privilege for AI Agents: Identity, Access and Tool Binding.” Microsoft (2026)
• “Least Privilege in AI Agents and Agentic Identity.” Permit.io (2026)
• “How to Establish Least Privilege for AI Agents and Assistants.” Zscaler (2026)
• “Top 10 for Agentic Applications.” OWASP (2026)
• “Agentic AI Governance for Enterprise Teams in 2026.” Himanshu AI (2026)
• “Five Engineering Patterns to Secure Agentic AI in 2026.” BayTech Consulting (2026)
• “Agent Incident Disclosure Standards.”Digital Applied (2026) 
• “Cloudflare Lets Users Create Permanent ID and a Wallet for AI Shopping.” Cloudflare (2026)
• Regulation (EU) 2024/1689. EU AI Act.
• NIST (2023). “AI Risk Management Framework 1.0.”

The Agent Registry Problem: Why You Cannot Govern What You Cannot See

AI Forensic Readiness: Evidence Architecture for the Agentic Enterprise

You’re Right: A Registry Isn’t Enough