Autonomy level
Read-only by default, write with approval for high impact, autonomous only for low-risk reversible actions.
Why. Matches authority to consequence.
Trade-off. More approvals reduce speed.
AI governance and assurance · AI agents · safety controls
AI agent safety architecture: tool permissions, sandboxing, approval gates, spend limits and a kill switch
How to let an AI agent act without letting it do harm: a permission broker issues narrow, short-lived tool access, risky code runs in a sandbox, high-impact actions wait for human approval, budgets cap spend and a kill switch stops everything.
A reference architecture from EITSAA, not a named client system. Service limits, prices and product features change, so confirm current details before you commit to a design.
- OWASP Top 10 for LLM Applications (excessive agency)
- NIST AI RMF
- Guidance for AI Adoption (human control)
- ISO/IEC 42001
- ACSC Essential Eight
- APRA CPS 230
Architecture
- People and clients
- AI and ML
- Security and identity
- Monitoring, audit and analytics
- SaaS and business systems
- Request or data flow
- Asynchronous or optional
- Boundary: region, account, network or trust zone
Request and data flow
A user, schedule or event gives the agent a task. The request carries identity so actions are attributed and limited to what that user may do.
Before using a tool, the agent requests a scope from the permission broker, which issues a narrow, short-lived credential bound to the task and user.
Every tool call goes through the tool gateway, which allows only registered tools, validates arguments against schemas and refuses anything else.
Code execution and browsing run in a sandbox with no standing secrets, restricted network egress and resource limits.
The gateway checks the budget: step count, tokens, tool calls and spend per task, user and day. Exceeding a limit stops the run.
Actions classed as high impact, such as payments, deletions or external messages, are proposed rather than executed, and wait for a human decision.
Approved calls reach business systems using the scoped credential, with read-only access by default and reversible changes where possible.
Approvals, rejections, reasons and the full action trail are written to the audit record, so each outcome can be reconstructed.
A kill switch can halt a single agent, a tool or a whole tenant within minutes, and is tested regularly.
Components and why they were chosen
| Component | What it does | Why this choice |
|---|---|---|
| Agent orchestrator (LangGraph, Bedrock Agents, Azure AI Foundry Agent Service, Semantic Kernel) | Plans and sequences steps. | Keeps reasoning separate from permissions. |
| Permission broker (cloud IAM, token exchange, secrets manager) | Issues scoped, short-lived credentials. | No standing privileges for agents. |
| Tool gateway (MCP gateway or API gateway) | Allow-list, schema validation and rate limits. | A single enforcement point for tools. |
| Sandbox (microVM, container with egress control) | Isolated execution. | Contains injected or buggy code. |
| Approval service | Human review with context. | Meaningful human control for consequential actions. |
| Trace store (OpenTelemetry, Langfuse) | Full step traces. | Investigation and evidence. |
Design decisions and trade-offs
Identity of the agent
Act on behalf of the user with a scoped token, never with a shared admin account.
Why. Preserves entitlements and attribution.
Trade-off. Token exchange adds complexity.
Where to validate
At the tool gateway, outside the model.
Why. The model cannot talk its way past code.
Trade-off. Schemas need upkeep.
Multi-agent design
Separate agents with separate scopes and a supervisor that cannot exceed the user's scope.
Why. Limits cascading errors.
Trade-off. More moving parts to trace.
Stop mechanism
Central kill switch plus per-run budgets.
Why. Contains loops and incidents fast.
Trade-off. Must be tested or it will not work when needed.
Well-architected pillars
Security
Least privilege, scoped short-lived credentials, sandboxing, egress control and untrusted-content handling for all tool outputs.
Reliability
Idempotent tool calls, retry with limits, checkpointed state and a manual fallback process.
Performance
Parallelise safe read calls and keep approval requests concise.
Cost optimisation
Budgets per task and day contain runaway loops and unbounded consumption.
Operational excellence
Runbooks for agent incidents, regular kill-switch drills and review of approval queues.
Sustainability
Cap step counts and prefer smaller models for routine steps.
Network, identity and data
Networking
The runtime and sandbox have no default internet access. Egress is allow-listed through a proxy. Tools are reached over private connectivity and the gateway is the only path from the agent to business systems.
Identity and least privilege
Each agent has its own identity. User-delegated actions use token exchange with narrow scopes. Approvers are separate from requesters. Administrative access to the broker and gateway is tightly restricted.
Data and encryption
Give agents the minimum data for the task. Mask sensitive fields in traces. Keep memory stores scoped and expire them. Classify tool outputs as untrusted content.
Security and compliance mapping
| Control | How this design addresses it |
|---|---|
| Excessive agency | Tool allow-lists, scoped credentials and approval gates address OWASP LLM06. |
| Human control | Approval gates and a kill switch support the Guidance for AI Adoption practice on human control. |
| Restrict administrative privileges | Agents never hold admin credentials. Essential Eight. |
| Application control | Only registered tools and signed agent versions can run. Essential Eight. |
| Operational resilience | Fallback procedures and tested stop mechanisms support CPS 230 expectations. |
Mappings show how the design supports each control. They are not certifications, and EITSAA does not claim IRAP assessment or compliance for any reference architecture.
Failure modes, targets and scaling
- Target RTOUnder one hour for the gateway and broker. Agents can be switched to manual process while recovering.
- Target RPONear zero for audit and approvals. Task state is checkpointed.
- Scaling and limitsSized by concurrent runs and tool call rates.
| Failure mode | Mitigation |
|---|---|
| Injected instructions in a document cause a harmful tool call | Treat content as data, validate calls at the gateway and require approval for high impact. |
| Agent loops and burns budget | Step, token and spend limits with alerts and automatic halt. |
| Approver clicks through without reading | Show concise context, sample decisions for audit and limit approval volume. |
| Tool schema change breaks behaviour | Contract tests for tools and staged rollout. |
| Credentials leak through traces | Mask secrets, never pass credentials through the model and scan stores. |
Observability and runbooks
- Steps, tool calls and spend per run.
- Approvals requested, granted and rejected.
- Blocked calls and schema violations.
- Kill-switch tests and time to halt.
Cost drivers and levers
Model tokens per step, sandbox compute, approvals handling time and trace storage.
- Hard budgets per run.
- Smaller models for routine steps.
- Cache read-only tool results.
- Sample traces for low-tier agents.
Implementation and fit
Phased approach
- Start with read-only tools and full tracing.
- Add the tool gateway, scoped credentials and budgets.
- Add sandboxing and approval gates for write actions.
- Add kill-switch drills, multi-agent supervision and regression tests for attacks.
Anti-patterns to avoid
- Giving an agent a shared admin credential.
- Validating tool calls only in the prompt.
- Approval with no context.
- No spend cap.
- An untested kill switch.
When to use it
- Agents that act on tickets, records, payments or messages.
- Multi-agent workflows that cross systems.
When not to
- A chatbot that only answers questions from approved documents.
- Processes that need deterministic, auditable logic only.
More in ai governance and assurance
ML · model risk · MLOps
ML model risk management and MLOps lifecycle: feature store, validation, registry, drift monitoring and challenger models
A governed lifecycle for scoring and prediction models: reproducible training from a feature store, independent validation including fairness and explainability, a registry with approval status, controlled deployment, drift monitoring and champion and challenger comparison.
Automated decisioning · human review
Automated decisioning platform: rules and models with explainability, human review and appeals
A decisioning pattern for decisions that affect people: rules and model scores combine in a versioned decision service, every outcome carries reasons, uncertain or adverse cases go to a trained reviewer and affected people have a clear route to challenge.
Evaluation · red-teaming · CI
AI evaluation and red-team pipeline: golden sets, adversarial tests, release gates and CI
A pipeline that turns testing into a release gate: golden sets and adversarial suites run in CI against every change, scores are compared with thresholds by risk tier, failures block promotion and results become evidence.
Related insights
Discuss this architecture
Tell us your systems, constraints and timeline. We reply within one business day, and the initial conversation is free.