Autonomy level
Tiered: read-only automatic, reversible with limits, irreversible with approval.
Why. Matches oversight to impact.
Trade-off. Approvals add latency, so tune thresholds with real data.
AI, LLM and real-time · AWS and Azure AI
AI agent with tool use and human approval
An agent that can do things, not just answer: a bounded planner, a registry of narrowly scoped tools, policy checks outside the model, human approval for consequential actions and a full action ledger.
A reference architecture from EITSAA, not a named client system. Service limits, prices and product features change, so confirm current details before you commit to a design.
- Voluntary AI Safety Standard
- AI Ethics Principles
- ACSC Essential Eight
- Australian Privacy Principles (APP 11)
Architecture
- People and clients
- AI and ML
- Security and identity
- Messaging, events and workflow
- Monitoring, audit and analytics
- SaaS and business systems
- Request or data flow
- Asynchronous or optional
- Boundary: region, account, network or trust zone
Request and data flow
The requester states a goal. The runtime attaches identity, scope and a conversation budget (steps, time and tokens).
The model proposes a plan and a tool call as structured output. It never holds credentials or calls systems directly.
Every proposed action goes to a policy engine that decides on the user's entitlements, the tool's risk tier and the arguments.
Read-only and reversible low-risk actions pass to the tool gateway immediately.
Consequential actions (payments, deletions, external emails, record changes above a threshold) pause in an approval workflow with the exact arguments shown.
The approver approves, edits or rejects. The decision, identity and time are recorded and the workflow resumes with a task token.
The gateway calls the business system using a narrowly scoped credential for that tool, with idempotency keys and timeouts.
Every tool call, result and decision is appended to the ledger, which feeds review, evaluation and incident response.
Components and why they were chosen
| Component | What it does | Why this choice |
|---|---|---|
| Agent runtime | Loop with step, cost and time limits, tool schema validation and memory scoped to the session. | Bounded loops prevent runaway cost and unintended chains of actions. |
| Foundation model with tool-use support | Plans and selects tools via structured calls. | Native function-calling gives typed, validatable output. |
| Policy engine (Cedar, OPA or AWS Verified Permissions) | Allow, deny or escalate each action by user, tool and arguments. | Authorisation lives outside the prompt, so injection cannot override it. |
| Tool gateway | Small, typed tools such as lookup_case or draft_email, each with its own credential. | Narrow tools beat a general HTTP tool. Least privilege per tool. |
| Approval workflow (Step Functions, Power Automate or ITSM) | Human-in-the-loop with timeouts and escalation. | Durable waiting with an audit trail. |
| Action ledger | Append-only record of plan, call, result and approver. | Provides evidence, replay and measurement. |
| Secrets Manager or Key Vault | Per-tool credentials with rotation. | The model never sees them. |
| Evaluation and red-team suite | Scenario tests including injection and misuse. | Agent behaviour must be tested like any critical software. |
Design decisions and trade-offs
Tools
Many small typed tools rather than a generic API caller.
Why. Smaller attack surface and clearer audit.
Trade-off. More tools to build and maintain.
Policy location
Deterministic policy engine outside the model.
Why. Security does not depend on model behaviour.
Trade-off. Policies need an owner and tests.
Memory
Session-scoped memory, no long-term personal memory by default.
Why. Limits privacy exposure and drift.
Trade-off. Less personalisation.
Well-architected pillars
Security
Policy outside the model, scoped credentials per tool, input and output guardrails, injection tests and approval for impact.
Reliability
Idempotent tools, step and time limits, retries with backoff and a safe failure state that hands over to a person.
Performance
Parallel read-only tool calls, streaming progress and caching of reference data.
Cost optimisation
Token and step budgets per request, smaller models for routing and larger ones for planning only when needed.
Operational excellence
Versioned prompts and tools, ledger-based review, weekly sampling of agent decisions and an emergency kill switch per tool.
Sustainability
Avoid unnecessary loops and prefer deterministic code where a rule will do.
Network, identity and data
Networking
Tool gateway in private subnets with egress only to approved business system endpoints. Model invoked over private endpoints. No tool can reach the internet unless explicitly defined.
Identity and least privilege
Act on behalf of the user using delegated tokens where the target system supports it. Otherwise a service identity with a policy that re-checks the user's entitlement. Approvers cannot approve their own requests.
Data and encryption
Minimise what is sent to the model. Redact personal information where it is not needed to decide. Ledger entries are retained per records policy and protected from alteration.
Security and compliance mapping
| Control | How this design addresses it |
|---|---|
| Human oversight | Approval step for consequential actions aligns with the Voluntary AI Safety Standard guidance on human control. |
| Restrict administrative privileges | Per-tool least privilege and no standing admin credentials. Essential Eight. |
| Logging | Complete action ledger supports ISM event logging and incident investigation. |
| Privacy | Data minimisation and purpose limitation support APP 3 and APP 6. |
Mappings show how the design supports each control. They are not certifications, and EITSAA does not claim IRAP assessment or compliance for any reference architecture.
Failure modes, targets and scaling
- Target RTOUnder an hour; disable by tool or globally with a kill switch
- Target RPOLedger near zero loss
- Scaling and limitsLimited by model and downstream system quotas. Design concurrency limits per tool.
| Failure mode | Mitigation |
|---|---|
| Prompt injection triggers a harmful tool call | Policy engine denies by user entitlement and risk tier. Retrieved content is data only. High-risk tools need approval. |
| Agent loops or burns budget | Hard step, time and token limits and alarms on cost per request. |
| Duplicate side effects on retry | Idempotency keys and a ledger check before every call. |
| Approver fatigue | Show exact arguments, batch low-risk items, track rejection rates and tune tiers. |
Observability and runbooks
- Per-tool success, error and approval rates.
- Ledger-based dashboards: actions by risk tier, approvals, overrides and time to approve.
- Evaluation suite run on every prompt, model or tool change.
Cost drivers and levers
Model tokens per step, number of steps, tool execution and approvals.
- Cap steps.
- Route with a smaller model.
- Cache reference lookups.
- Replace repeated reasoning with deterministic automation.
Implementation and fit
Phased approach
- Start with read-only tools and a narrow task.
- Add reversible write tools with limits and the ledger.
- Introduce approvals for impactful actions.
- Expand scope once measured error and override rates are acceptable.
Anti-patterns to avoid
- Giving the agent a database admin credential.
- One generic HTTP tool.
- Relying on the system prompt for security.
- No ledger or kill switch.
When to use it
- Multi-step processes across systems with clear rules.
- Triage, drafting, data gathering and preparation of decisions.
- Teams ready to run an evaluation process.
When not to
- Fully autonomous decisions that affect people's entitlements.
- Processes with no clear policy to encode.
- Anything that can be solved by a simple workflow.
More in ai, llm and real-time
WebRTC · LiveKit
Real-time video and voice for iOS and Android with LiveKit on AWS
A WebRTC platform for mobile apps built on the LiveKit SFU: token-based access, UDP and TURN/TLS connectivity, server-side recording to S3, webhooks, ingress for streams and scaling across nodes and regions.
WebRTC · Amazon Chime SDK
Meetings, voice and recording with the Amazon Chime SDK
A managed real-time platform on AWS: a Meetings API backend, mobile and web clients, media regions close to users, SIP media applications for phone calls and media pipelines for recording, all driven by Lambda and EventBridge.
Google Cloud
LLM inference on Google Cloud with Vertex AI, Cloud Run and VPC Service Controls
A private, governed Gemini and Model Garden inference platform: Cloud Armor at the edge, Cloud Run for the application, Vertex AI over Private Service Connect, Pub/Sub for async jobs and BigQuery for audit and cost analytics.
Related insights
Discuss this architecture
Tell us your systems, constraints and timeline. We reply within one business day, and the initial conversation is free.