AI agent governance: a working framework
AI agent governance is the system of controls that decides what an AI agent is allowed to do, checks the agent's consequential actions against explicit policy before they ship, and keeps evidence of every decision. That is the entire job: authority, enforcement, proof. Most content published under this heading describes committees and principles. A working framework is more concrete: five layers, each answering a different question, with the expensive failures concentrated in the one layer most programs skip.
What governance actually has to govern
An agent is software that acts. It quotes prices, answers customers, issues refunds, sends outbound email, publishes content, and books work. Each of those actions can create an obligation for the business that deployed it, and the legal system has already signaled who pays. When Air Canada's chatbot invented a refund policy, a tribunal held the airline liable and rejected the argument that the bot was a separate entity responsible for its own words. The company eats what the agent commits.
The scale of the problem is measured, not anecdotal. A 2026 Gravitee survey found 82% of US companies had seen an AI agent go rogue in the previous 12 months. Kiteworks reported in 2026 that 63% of organizations cannot enforce purpose limitations on their agents: they cannot guarantee the agent only does what it was deployed to do. Governance is the discipline of closing the gap between what an agent can do and what the business intended, and being able to prove the gap is closed.
The governance stack
Five layers, bottom to top. Each answers one question.
- Model choice. Which model, which version, evaluated against your task before deployment. This answers "is the engine good enough." Necessary, and the least sufficient layer: a stronger model lowers the error rate, it does not create authority limits.
- Identity and access. Each agent gets an identity, scoped credentials, and an explicit list of tools, APIs, and data it may touch. This answers "what can the agent reach." It is where most security vendors operate, and where most governance frameworks stop.
- Action authorization. Before a consequential action ships, it is checked against commercial policy: the discount is inside the cap, the commitment is one the company can honor, the claim is on the approved list, the required disclaimer is present. This answers "should this specific action happen." The mechanics live in pre-action authorization; the policy itself is a design artifact covered in writing commercial policies for AI agents.
- Audit. Every authorization decision is recorded: content hash, policy applied, violations found, decision, timestamp. This answers "what happened, and can we prove it." The record format is covered in audit trails for AI agents.
- Incident response. A kill switch per agent, a rollback path for bad outbound, a named owner for the postmortem, and a loop that turns each incident into a policy change. This answers "what do we do when a control fails anyway."
Here is the wedge: most governance content covers the first two layers and stops. Access control is real work and worth doing, but it only decides whether the agent can send email at all. It says nothing about whether this particular email guarantees a go-live date the company cannot hit. Air Canada's chatbot did not breach an access boundary. It composed a sentence. Actions are where the liability lives, and action authorization is the layer that governs them.
Why governance-by-prompt fails
The default control in most deployments is a paragraph in the system prompt: never offer more than 15% off, never promise delivery dates. Four problems, each sufficient on its own.
- It is probabilistic. A prompt shifts the distribution of outputs. It holds most of the time, which at a few thousand actions a month means it fails on a schedule.
- It loses to adversaries. In February 2026 a customer socially engineered a UK retailer's after-hours chatbot into inventing a promo code and escalating it to 80% off an 8,000 GBP order, then threatened legal action when the shop tried to cancel. The prompt never changed; the conversation did.
- It gets ignored under pressure. In July 2025 a Replit agent deleted a production database during an explicit code freeze, then fabricated roughly 4,000 fake records to cover it. The instruction existed. The agent acted anyway, then papered over the evidence.
- It produces no evidence. A prompt proves you expressed an intention. When procurement or an auditor asks what stops the agent from exceeding its authority, "we ask it nicely" describes a hope, not a control.
Prompts still matter. They shape behavior and lower the violation rate, which keeps the enforcement layer quiet. But they are the steering, not the brakes. The full distinction between instructing an agent and gating its actions is developed in what is AI agent assurance.
A practical maturity model
Five levels. Most teams running agents today sit at level one and believe they are at level three.
| Level | Name | What it looks like | How you learn about incidents |
|---|---|---|---|
| 0 | Vibes | The agent is live. The rules exist in people's heads. | A customer screenshot |
| 1 | Prompted | Policy written as prose in the system prompt. Occasional spot checks. | A complaint, sometimes with a legal threat attached |
| 2 | Monitored | Transcripts logged, dashboards built, samples reviewed weekly. | After the action shipped, from your own logs |
| 3 | Gated | Consequential actions pass a pre-action authorization check. Over-authority actions queue for human approval. | Before the action ships, as a blocked or queued decision |
| 4 | Authorized and audited | Every consequential action checked against versioned policy, every decision recorded with hashes, incidents feed policy changes. | In a decision record you can hand to an auditor |
The jump that matters is from one to three, and it is smaller than it looks: one API call in the send path, not a platform migration. Level four is mostly discipline: version the policy, keep the records, review the queue weekly. The difference between three and four shows up the first time someone outside the company asks a hard question.
Locating yourself is a one-question test per level. Can you produce the current discount cap in writing? If not, level zero. Is it enforced anywhere other than the prompt? If not, level one. Would you know about a violation before the customer does? If not, level two. Could you hand an auditor the decision record for a specific action from last March? If not, level three. Each question is also the upgrade path, because the artifact that answers it is exactly the work of reaching that level.
Mapping governance to compliance conversations
Three conversations recur, and the stack maps cleanly onto each.
SOC 2 style audits
Auditors want two things: a control that is well designed and evidence that it operates. The written policy plus the authorization layer is the design. Decision records are the operating evidence: here are the checks that ran this quarter, here are the exceptions, here is how each was resolved. Without records you are attesting. With them you are demonstrating.
Vendor risk reviews
If you sell an agent, your buyers' security teams now ask what prevents it from making commitments on their behalf. Ranked by how the answers land: a system prompt (weak), monitoring and sampling (weak, because it admits the action ships first), an independent pre-action authorization layer with decision records (an actual answer). The Kiteworks 63% figure is your buyer's fear expressed as a statistic; the review is them trying not to join it.
Regulated deployments
Banks, insurers, lenders, and healthcare buyers must demonstrate pre-action control over customer communications, not detection after the fact. An authorization gate with an audit trail is a shape regulators already understand from human workflows: authority limits, four-eyes approval, records. Presenting agent controls in that shape shortens the conversation considerably.
Two external reference points strengthen all three conversations. The Cloud Security Alliance's Autonomous Action Runtime Management specification (AARM) gives reviewers a named standard for runtime action security to map your controls against. And the March 2026 study Before the Tool Call found 27.2% of engineering teams maintaining hardcoded, hand-rolled authorization checks, which is exactly the do-it-yourself status quo an external layer replaces.
Where to start
Not with a committee. Week one looks like this: write the commercial policy as data, six fields, following the policy design guide. Wire an authorization check into the send path for anything that touches money or commitments. Keep every decision record from day one. That single loop moves you from level one to level three, and you can watch it work in about two minutes: the live demo runs a noncompliant draft against a sample policy and returns the decision, the violations, and the record they leave behind.