An agent that can reason about an action and an agent that is permitted to take it are two different things. Treating them as one is how capable systems end up with authority nobody deliberately gave them.
Proposing is not the same as being permitted
An LLM can produce a plan or a tool request. Whether that request is carried out is a separate decision. If the two are not separated, the agent's effective authority becomes whatever its output and tool wiring allow, which is rarely what anyone intended.
Risk assessment
Risk can be derived from properties of the action and its context: whether it is reversible, how wide its impact could be, which environment it targets, how sensitive the data involved is, and how strong the supporting evidence is. Ideally these properties come from action metadata and deterministic rules, not from the model grading its own proposal.
Deterministic policy
A policy layer takes the proposed action and its risk and returns a decision: allow, require approval or deny. Deterministic means the same inputs give the same outcome, so policy can be tested, reviewed and explained. Leaving that decision to model output alone makes authority depend on wording and context in ways that are hard to audit.
| Action type | Illustrative policy outcome |
|---|---|
| Read-only query | Allow |
| Reversible change in a non-production environment | Allow, with logging |
| Reversible change in production | Require human approval |
| Irreversible or high-impact change | Deny, or require explicit senior approval |
Illustrative shape only. Real policies depend on the organisation and the system.
Human approval and auditability
When approval is required, the approver should see the proposed action, the evidence behind it, the assessed risk and the reason policy asked for approval. Approval should be bound to that specific action rather than granted broadly. Every step, from proposal to outcome, should be recorded so the decision can be reconstructed later.
Trade-offs
Stricter boundaries slow things down and add review work. A sensible approach is to begin conservatively and widen autonomy only as observed behaviour justifies it, keeping the ability to tighten again.
Key takeaways
- Proposing an action and being authorised to execute it are different questions.
- Risk and policy should be computed by deterministic, testable logic outside the model.
- Approvals should be specific, informed and recorded.
- Start with narrow authority and widen it based on evidence.