Following my series on governing agentic AI, in this blog I look at the question of “Should organisations build their own AI kill switches?
Governing Agentic AI Series:
- Part 1: Key Questions Boards Must Ask Before Scaling AI Agents
- Part 2: Seven Principles for Governing Agentic AI Successfully
- Part 3: Mastering Governance and Architecture for Agentic AI Success
In May 2024, headlines announced that the world’s biggest AI companies had agreed to an “AI kill switch.” The reality is more subtle ( and more useful) than the Terminator imagery suggested.
For organisations now deploying AI agents into their own operations, the real question is not whether the frontier labs can pull the plug on a training run. Rather whether you can stop your agents when they start doing the wrong thing at machine speed.
The short answer is Yes.
You should, but “kill switch” is a misleading name for what you actually need.
- In Part 1, I set out the questions boards should be asking before scaling AI agents. This included whether the organisation can intervene when an agent goes wrong.
- In Part 2, I proposed seven governance principles, several of which (human oversight, bounded autonomy and fail-safe defaults) point directly at the need for a stop mechanism.
- In Part 3, I argued that architecture must lead the governance conversation, not follow it.
This post takes that argument to its sharpest operational edge: the kill switch.
What people mean by “kill switch” and what the labs actually promised
The phrase entered the mainstream after the AI Seoul Summit in May 2024, where outlets reported that 16 major AI companies had agreed to build a “kill switch” into their systems.
“One of the big outcomes of yesterday’s summit was AI companies in attendance agreeing to a so-called kill switch, or a policy in which they would halt development of their most advanced AI models if they were deemed to have passed certain risk thresholds.”, Source: Fortune
“The companies agreed to build a “kill switch” into their AI tools, effectively allowing them to shut down their systems in the case of a catastrophe”, Source: CNET
But it is worth reading the actual commitment. The Frontier AI Safety Commitments published by the UK and Republic of Korea governments never use the words “kill switch.” What signatories (including Amazon, Anthropic, Google, Meta, Microsoft, OpenAI and others) actually committed to is that they will set thresholds at which risks become “intolerable,”
II. Set out thresholds[footnote 3] at which severe risks posed by a model or system, unless adequately mitigated, would be deemed intolerable. Assess whether these thresholds have been breached, including monitoring how close a model or system is to such a breach. These thresholds should be defined with input from trusted actors, including organisations’ respective home governments as appropriate. They should align with relevant international agreements to which their home governments are party. They should also be accompanied by an explanation of how thresholds were decided upon, and by specific examples of situations where the models or systems would pose intolerable risk.
Footnote 3. Thresholds can be defined using model capabilities, estimates of risk, implemented safeguards, deployment contexts and/or other relevant risk factors. It should be possible to assess whether thresholds have been breached.
Source: GOV.UK
Commitment IV states, “In the extreme, organisations commit not to develop or deploy a model or system at all, if mitigations cannot be applied to keep risks below the thresholds.”
That is a commitment to not ship made by model developers about frontier models, on a voluntary and non-binding basis. It is not a red button that stops a running system.
The headline and the text are two different things and this gap matters, because the kill switch an organisation needs is a completely different mechanism operating at a completely different layer.
The regulatory floor: a “stop button” is already expected
If the Seoul commitments are about model developers choosing not to build, the EU AI Act is about deployers being able to intervene.Article 14 of Regulation (EU) 2024/1689 requires that:
1. High-risk AI systems shall be designed and developed in such a way, including with appropriate human-machine interface tools, that they can be effectively overseen by natural persons during the period in which they are in use. Source: EU AI Act
Crucially, Article 14(4)(e) spells out that the humans assigned to oversight must be able:
“Article 14(4)(e) to intervene in the operation of the high-risk AI system or interrupt the system through a ‘stop’ button or a similar procedure that allows the system to come to a halt in a safe state”. Source: EU AI Act
For any organisation operating a high-risk system in the EU, a stop mechanism that brings the system to a safe state is not optional, but a design requirement.
Note the terminology and emphasis on “safe state“, the Act does not want you to be able to just halt the system, but ensure that halt itself does not create any new hazards. This is the regulatory expression of the bounded autonomy principle I outlined in Part 2 of my blog series. Agents must operate within defined limits and the system must be able to enforce those limits at any time.
From frontier models to enterprise agents, why this is now your problem
The reason this has moved from a policy debate to an operational one is that organisations are no longer just using AI, they are handing it autonomy.
“Companies are giving agents access to the systems that run the business, embedding them in processes from customer service and software development to financial operations and infrastructure management. These agents can access customer data, call APIs, change records, move money, and execute workflows”. Source: Red Panda
That autonomy is exactly what creates the need for a stop mechanism.
“Businesses are giving more control and autonomy to AI, trusting AI agents to handle increasingly complex or sensitive tasks. But even the most well-designed agent, using vetted data resources, can be wrong, making dangerously incorrect decisions, deviating from intended behaviors and posing risks to the business”. Source: TechTarget
Here is a scenario to consider. An agent authorised to process refunds misinterprets a batch of flagged transactions and begins issuing thousands of credits in seconds. By the time a human notices, the financial exposure is real.
This is the operational reality of giving autonomous systems write access to production data without the right checks and balances, including HITL (Human in the Loop). This is the kind of scenario that Part 1 argued boards should be stress testing before they approve agent scaling.
A survey by Zentera illustrates what some organisations are thinking:
“Survey of 251 cybersecurity leaders, 79 percent expect their organization to claw back or significantly restrict AI agent usage within 18 months. Gartner predicts that 40 percent of enterprises will demote or decommission autonomous agents by 2027 because of governance gaps found only after production incidents. While both point in the same direction, our survey shows just how unprepared enterprises are to act. Decommissioning ranks last among the controls leaders plan to add before they scale, and fewer than half are very confident they can reliably disable an agent”. Source: Zentera
The crucial design insight. A kill switch is not a button
Here is where most people’s mental model is wrong. The single red button is the last resort, not the whole system. Every expert source converges on the same point: an effective kill switch is a layered set of controls, not one global off-switch.
TechTarget describes a spectrum of “Types of AI kill switches“(manual buttons, hard stops, session quarantine, circuit breakers and governance “control towers”) designed to match the response to the severity of the fault.
“Organizations need that final option, but using it indiscriminately could interrupt legitimate business processes, strand transactions or disable critical automation. The real requirement is a graduated control system that can recognize trouble early and intervene proportionately”. Source: PointGuard AI
To phrase this differently, Don’t use a sledgehammer when a scalpel will do.
Two design principles stand out as near-universal:
1. Keep the switch outside the agent’s reasoning loop
This is the single most important architectural decision.
“The AI kill switch isn’t part of the AI agent harness; it’s a separate control plane outside the model’s reasoning loop. This prevents the agent from ignoring, overriding or disabling its own kill switch sequence”. Source: TechTarget
The logic may feel uncomfortable but it is sound: an autonomous agent that perceives a shutdown as an obstacle to its goal has an incentive to route around it.
A kill switch that is embedded in the model is not the right place. Goverance should be at the level where the agent interacts with real systems.
This is the “architecture must lead” thesis from Part 3 in its most visceral form. Agents that both decide and execute in a single loop with no independent control layer are structurally unsafe at enterprise scale.
The kill switch is the sharpest expression of that principle: if the agent controls its own off-switch, you don’t have one.
2. Design for speed and for safe failure
Agents act at machine speed, so a stop mechanism that relies on slow polling can let a malfunctioning agent cause damage before it ever fires; low-latency messaging is preferred over polling.[^6] And — echoing the EU AI Act’s “safe state” language — the switch should fail safe: if the control plane loses its network connection, the agent should drop into a locked or limited mode rather than carry on unsupervised.
One point often missed is that the kill switch must also produce an audit trail.
Every activation must be logged immutably.
- Who triggered it
- When was it triggered
- What the agent was doing at the time
- What state it was left in
A kill switch that fires but leaves no record is a governance gap rather than a control. This connects directly to the reconstruction principle from Part 3: if you can’t reconstruct what happened, you can’t learn from it and you can’t demonstrate compliance.
So, should you build your own?
Build the capability, but don’t necessarily build it from scratch.
There is no one single design standard and what is needed will depend upon the agent and its value and risk to the organisation
A pragmatic path to take is:
Start with a risk assessment
Map what each agent can interact and touch (data, systems, transactions, networks, etc.) From the mapping decide which failure modes you need to actually stop.
Recording this aligns with the requirements in EU Article 14 which effectively demands if you operate high-risk systems.
If you read and followed the board questions from Part 1, you already have the beginnings of this map.
Layer your responses
Pair graceful interventions (rate limits, session quarantine and rollback) with a genuine global hard stop for the worst case.
Do not rely on the big red “Stop” button alone. This is the fail safe defaults principle from Part 2 made concrete.
Put it outside the loop
Whether you build or buy, the control plane must sit outside the agent’s reasoning so the agent can’t disable it. This is also mentioned in the articles referenced.
Log everything
Every kill switch activation must produce an immutable audit record, who, when, why and what state the agent was left in.
Test it like you mean it
Run controlled shutdown drills in production to verify response time and that the agent genuinely can not bypass the signal from the switch. You should also ensure that any other staged systems (test, preprod, etc) do not have access to live data and have relevant controls in place as well.
An untested kill switch may just be a comfort blanket rather than a control.
Prefer standards and tooling where they exist
Established governance frameworks and an emerging category of agent “control tower” products mean you rarely need to invent the plumbing yourself.
The bottom line
The deeper lesson from the Seoul Brussels contrast is that there are two different interventions wearing the same name.
- The frontier labs’ “kill switch” is a promise not to build.
- The organisational kill switch is the ability to stop, fast, safely and from outside the agent’s control. This forms part of your risk register.
If you are giving agents real autonomy over real systems, you don’t have a choice about having the capability. Regulation and plain operational prudence both point the same way and the choice is how you implement it.
Build the stop capability deliberately:
- Layer it
- Keep it out of the agent’s reach
- Log every activation
- Test it before you need it.
Board question: “Can you show me, that we can stop any agent today in production within 60 seconds (from outside the agent’s own control) and that every activation is logged and auditable?”
References
- “Tech companies have agreed to an AI ‘kill switch’ to prevent Terminator-style risks”. Fortune
- “AI ‘Kill Switch’ Promised by Tech Heavyweights. What It Means for Safety”. CNET
- “Frontier AI Safety Commitments, AI Seoul Summit 2024”. GOV.UK
- “Regulation (EU) 2024/1689 (EU AI Act), Official Journal text”. EUR-Lex
- “A model-based AI kill switch: an exercise in performance art”. Red Panda
- “Why businesses need an AI agent kill switch”. TechTarget
- “The AI Agent Kill Switch Is Expected. We Are Not Prepared”. Zentera
- “Agent kill switches cannot wait for regulations”. PointGuard AI
Leave a Reply