Agent versus workflow
Deterministic workflows for known processes, agents for open-ended steps within them.
Why. Predictability where it matters and flexibility where it helps.
Trade-off. More design effort than a single general agent.
AI, LLM and real-time · AI platform
AI platform for LLM bots and agents integrated with business processes
A reusable platform, not a single bot: channels, an agent runtime, a model gateway, retrieval, a tool registry with policy, a durable workflow engine, human approvals, CRM and ERP connectors, evaluation and cost control.
A reference architecture from EITSAA, not a named client system. Service limits, prices and product features change, so confirm current details before you commit to a design.
- Voluntary AI Safety Standard
- AI Ethics Principles
- Australian Privacy Principles (APP 11)
- ACSC Essential Eight
Architecture
- People and clients
- Network and edge
- AI and ML
- Security and identity
- Messaging, events and workflow
- SaaS and business systems
- Monitoring, audit and analytics
- Request or data flow
- Asynchronous or optional
- Boundary: region, account, network or trust zone
Request and data flow
A user writes in a channel. The gateway authenticates, resolves entitlements, redacts personal information the agents do not need and enforces rate limits.
The runtime classifies intent and chooses the right agent or a deterministic workflow. Simple, rule-based requests never need a model.
Retrieval fetches only documents the user may access, with citations and freshness metadata.
Reasoning calls go through a model gateway that selects a model by task, applies budgets and caching, enforces guardrails and records tokens per team and use case.
When the agent proposes an action, the tool registry validates the call against a typed schema and the policy engine decides allow, deny or escalate by user, tool, arguments and risk tier.
Long or consequential processes run in a durable workflow engine, which provides timers, retries, compensation and an audit trail. The agent starts and monitors them but does not hold their state.
Approval steps appear in an inbox or Teams card with the exact arguments. The approver can approve, edit or reject, and the decision resumes the workflow.
Connectors call CRM, ERP and ITSM APIs with per-tool credentials, idempotency keys and timeouts. Results return to the agent and the user.
Every step emits traces. Evaluation jobs replay golden scenarios on each change and sample live conversations for review.
Components and why they were chosen
| Component | What it does | Why this choice |
|---|---|---|
| Channel gateway | Authentication, redaction, throttling and channel adapters (web, Teams, email, voice). | One control point for every entry. |
| Agent runtime (LangGraph, Semantic Kernel, Bedrock Agents or Azure AI Agent Service) | Planning, tool selection, memory and step limits. | Use a framework for orchestration, but own the policy and state outside it. |
| Model gateway (APIM, LiteLLM-style proxy or a managed gateway) | Routing, fallbacks, quotas, caching and cost attribution. | Model choice becomes configuration, not code. |
| Retrieval service | Hybrid search, permission filtering, citations. | Shared by all agents to keep the permissions model consistent. |
| Tool registry (MCP servers or OpenAPI) with a policy engine | Typed tools, risk tiers and authorisation (Cedar, OPA). | Prevents agents holding open-ended access. |
| Workflow engine (Temporal, AWS Step Functions, Camunda) | Durable execution, timers, human tasks and compensation. | Business processes outlive a chat session. |
| Approval inbox (Teams, ServiceNow or a console) | Human oversight with evidence. | Accountability for consequential actions. |
| Connectors (Salesforce, Dynamics, SAP, ServiceNow, Xero) | Scoped integration with business systems. | Integration is where agents deliver value. |
| Evaluation and observability (OpenTelemetry, Langfuse or similar, cloud-native monitoring) | Traces, golden sets, LLM-as-judge with human calibration. | Quality becomes measurable and regression-tested. |
| Cost controls | Budgets per team, token quotas, caching and model routing. | Prevents surprise bills. |
Design decisions and trade-offs
State
Workflow state in a durable engine, conversation memory in a store, nothing in prompts alone.
Why. Processes survive restarts and are auditable.
Trade-off. Another platform component.
Tools
Registry of small typed tools with per-tool credentials and risk tiers.
Why. Least privilege and clear audit.
Trade-off. Tool catalogue to maintain.
Model access
Everything through the model gateway.
Why. Cost, safety and portability.
Trade-off. A critical component that must be highly available.
Quality
Evaluation as a release gate.
Why. Agents regress silently.
Trade-off. Needs curated scenarios and reviewer time.
Well-architected pillars
Security
Policy outside the model, scoped tool credentials, permission-aware retrieval, guardrails, injection testing and approval for impact.
Reliability
Durable workflows, idempotent tools, fallbacks across models and safe hand-over to people.
Performance
Streaming, parallel read-only tool calls, caching and small models for routing.
Cost optimisation
Per-team budgets, caching, model routing and caps on steps and tokens.
Operational excellence
Versioned prompts, tools and policies, canary releases, evaluation gates and incident runbooks including a kill switch per tool.
Sustainability
Prefer deterministic code, smaller models and batching where latency allows.
Network, identity and data
Networking
Platform components in private subnets or a private environment. Model, retrieval and connector traffic uses private endpoints. Egress is limited to approved business system endpoints and model APIs.
Identity and least privilege
User identity propagates to retrieval and, where supported, to tool calls using delegated tokens. Otherwise a service identity re-checks the user's entitlement in the policy engine. Approvers cannot approve their own requests. Workload identities replace keys.
Data and encryption
Classify data at ingestion and in prompts. Redact personal information not needed for the task. Define retention for conversations, traces and approvals. Keep an immutable action ledger. Do not train on customer data without a clear basis.
Security and compliance mapping
| Control | How this design addresses it |
|---|---|
| Human oversight | Risk-tiered approvals follow the Voluntary AI Safety Standard guidance on human control. |
| Restrict administrative privileges | Per-tool least privilege and no standing admin credentials. Essential Eight. |
| Logging | Traces and the action ledger support ISM event logging and incident investigation. |
| Privacy | Minimisation and purpose limitation support APP 3, 6 and 11. |
Mappings show how the design supports each control. They are not certifications, and EITSAA does not claim IRAP assessment or compliance for any reference architecture.
Failure modes, targets and scaling
- Target RTOUnder an hour for the platform, with workflows resuming from durable state
- Target RPONear zero for workflow state and the ledger
- Scaling and limitsBounded by model quotas and connector API limits. Design concurrency limits per tool and per team.
| Failure mode | Mitigation |
|---|---|
| Agent loops or overspends | Step, time and token budgets with alarms. |
| Harmful tool call from injected content | Policy engine by entitlement and risk tier, approval for impactful actions. |
| Workflow step fails halfway | Durable retries and compensation actions defined per step. |
| Model provider outage | Gateway fallbacks and graceful degradation to search or human queues. |
| Quality regression after a prompt change | Evaluation gate blocks release, canary with rollback. |
Observability and runbooks
- Traces per conversation across agent steps, retrieval, model calls and tools.
- Quality dashboards from evaluation sets, user feedback and sampled review.
- Cost and usage by team, agent and tool, with budget alerts.
- Runbooks: kill switch per tool, model failover, workflow replay.
Cost drivers and levers
Model tokens, retrieval infrastructure, workflow engine, connector licences and reviewer time.
- Route simple tasks to small models or rules.
- Cache retrieval and answers.
- Cap steps.
- Retire low-value agents on evidence.
Implementation and fit
Phased approach
- Pick one process with a clear owner and measurable outcome.
- Build the gateway, retrieval, one agent, one workflow and the ledger.
- Add approvals, evaluation gates and cost controls.
- Open the platform to further teams through the tool registry.
Anti-patterns to avoid
- One mega-agent with every tool.
- Business logic in prompts.
- No evaluation set.
- Direct model keys in each team's code.
- Autonomy for decisions that affect people's entitlements.
When to use it
- Organisations planning several AI use cases across teams.
- Processes spanning CRM, ERP and service desk.
When not to
- A single simple chatbot, where a managed bot is enough.
- No process owner or measurable outcome.
More in ai, llm and real-time
AI gateway · multi-cloud
Sovereign LLM gateway for government: one policy point across Bedrock, Azure OpenAI, Vertex AI and self-hosted models
A central AI gateway that every agency application calls instead of a model provider. It authenticates the caller, applies policy and PII redaction, routes each request to an approved Australian-region model, enforces budgets and writes a tamper-evident audit record.
AWS · Kubernetes · vLLM
Self-hosted open-weight LLM inference on Kubernetes with GPU nodes and no internet egress
A private inference service running open-weight models with vLLM on EKS GPU nodes: queue-aware autoscaling, a governed model registry, weights loaded from encrypted S3 over a gateway endpoint, and a VPC with no route to the internet.
Agentic AI · MCP
Enterprise agentic AI platform: multi-agent orchestration, MCP tool gateway and governed tool identities
A platform for agents that act in enterprise systems: a supervisor and specialist agents, an MCP gateway that holds every tool credential, a policy engine deciding each call, human approval for consequential steps, continuous evaluation and red-teaming, and a complete audit trail.
Related insights
Discuss this architecture
Tell us your systems, constraints and timeline. We reply within one business day, and the initial conversation is free.