AI, LLM and real-time · AI gateway · multi-cloud
A central AI gateway that every agency application calls instead of a model provider. It authenticates the caller, applies policy and PII redaction, routes each request to an approved Australian-region model, enforces budgets and writes a tamper-evident audit record.
A reference architecture from EITSAA, not a named client system. Service limits, prices and product features change, so confirm current details before you commit to a design.
An agency application calls the gateway with a short-lived workload token (OIDC federation from Entra ID or the agency IdP). Applications never hold provider API keys, so rotating or revoking a model is a gateway change, not an application release.
The gateway asks the policy engine whether this application, for this registered use case and this data classification, may use this capability. Policy is code in a versioned repository, so changes are reviewed and testable.
The request body is inspected for personal and sensitive information (names, addresses, Medicare and tax file numbers, case identifiers). Depending on policy it is blocked, masked or replaced by reversible tokens held in the gateway.
The sanitised prompt goes to the router together with the data classification and the use case's approved model list.
For general workloads the router sends to Amazon Bedrock in Sydney over an interface VPC endpoint. Bedrock does not use customer prompts to train base models, and the endpoint policy limits which models can be called. Check whether each model is served in-region or through a cross-region inference profile.
For use cases bound to Microsoft tooling, the router calls an Azure OpenAI regional deployment in Australia East over a private endpoint. Use regional (Standard) deployments and confirm the data processing location for the chosen model.
Where a model is only offered on Google Cloud, the router calls Vertex AI in an Australian region through Private Service Connect. Model availability by region differs, so the routing table is checked against the provider's current regional list.
Higher-classification or high-volume workloads go to open-weight models on the agency's own GPU nodes with no provider hop at all. The router prefers this tier when the policy says data must not leave the account.
Every request writes a signed audit record (caller, use case, policy decision, model, token counts, redaction actions, response hash) to an Object Lock bucket. Prompt text is stored only if policy requires it and is encrypted with a separate key.
| Component | What it does | Why this choice |
|---|---|---|
| AI gateway (Kong, Envoy AI Gateway, LiteLLM Proxy, Azure API Management or API Gateway) | Single OpenAI-compatible entry, authentication, quotas, streaming and request IDs. | Standardises the API so models can change without application change. |
| Policy engine (OPA or Cedar) | Decides allow, deny, redact or downgrade by caller, use case, data class and model. | Policy as reviewed code, with decision logs for audit. |
| PII redaction (Amazon Comprehend, Azure AI Language, Microsoft Presidio or a custom detector) | Finds and masks personal information in prompts and responses. | Data minimisation before data reaches any provider. |
| Model router | Chooses provider by classification, capability, cost, latency and health, with fallback and semantic cache. | Avoids lock-in and gives sovereignty a control point. |
| Amazon Bedrock with Guardrails | Managed foundation models in Sydney with content and topic filters. | Fast access to current models with no GPU operations. |
| Azure OpenAI Service | Regional deployments in Australia East. | Required where agencies standardise on Microsoft. |
| Vertex AI | Gemini and partner models in Australian Google Cloud regions. | Capability coverage and negotiation leverage. |
| Self-hosted open-weight models (vLLM) | Full control of weights, data path and cost at volume. | Meets the strictest residency and classification needs. |
| Audit store (S3 Object Lock, CloudTrail Lake or a SIEM) | Immutable decision and usage records. | Supports investigation, FOI and cost allocation. |
Gateway or direct SDK calls
Why. One place for policy, redaction, budgets and evidence.
Trade-off. The gateway becomes critical infrastructure and needs multi-AZ capacity and its own on-call.
Where to redact
Why. Agencies cannot trust every application team to redact consistently.
Trade-off. Detection is imperfect. Treat redaction as a layer, not a guarantee, and test with agency-specific identifiers.
Routing basis
Why. Sovereignty and classification are constraints. Cost is an optimisation inside them.
Trade-off. Smaller candidate sets for sensitive data mean less choice and sometimes weaker models.
Network path to providers
Why. Removes the internet from the data path and simplifies egress approval.
Trade-off. Endpoint and DNS design per provider, and some features arrive on public endpoints first.
Prompt logging
Why. Logs themselves become a store of sensitive data and a records and FOI subject.
Trade-off. Less content makes incident analysis harder. Allow time-boxed content capture for investigations.
Workload identity, deny-by-default policy, private endpoints, KMS keys per classification, endpoint policies that name allowed models, WAF in front of any public entry, and no provider keys in application code.
Stateless gateway across three AZs, health-checked provider pools, circuit breakers and fallback to a second approved provider where classification allows, and graceful degradation to queued or human handling.
Streaming responses, connection pooling, exact and semantic caching for repeated prompts, and region-local providers to keep latency low.
Per-team budgets and token quotas, model tiering by task, caching, committed throughput only after measured baselines, and showback reports.
Gateway and policy deployed from Git with canary releases, synthetic probes per provider, and a kill switch per model and per use case.
Route routine tasks to small models, cache aggressively and scale self-hosted GPU capacity to demand.
The gateway lives in private subnets across three AZs behind an internal load balancer reached over Direct Connect, Site-to-Site VPN or Transit Gateway from agency networks. Interface endpoints reach Bedrock, Secrets Manager and KMS. Azure OpenAI and Vertex AI are reached over private connectivity where the agency has it, otherwise through a single egress path with FQDN allow-listing and TLS inspection. No other internet egress is allowed.
Applications authenticate with workload identity federation (OIDC) and receive scoped tokens carrying application, use case and classification claims. The gateway uses its own roles per provider with least-privilege, short-lived credentials. Administrators use phishing-resistant MFA and just-in-time elevation, and policy changes require two reviewers.
Classify the data before it reaches the gateway and carry the label as a claim. Keep prompts and responses inside Australian regions, record where each provider processes data, and retain audit records under the applicable records authority. Do not allow providers to retain prompts beyond abuse monitoring where an opt-out is available, and document any residual retention in the privacy impact assessment.
| Control | How this design addresses it |
|---|---|
| Restrict administrative privileges | Just-in-time admin, separate break-glass, no standing provider keys. Essential Eight. |
| Application control and patching | Immutable container images, signed builds and a monthly patch cadence for the gateway. Essential Eight. |
| Event logging and monitoring | Signed audit records and policy decision logs support ISM event logging guidance and agency SIEM ingestion. |
| Data residency and sovereignty | Routing policy restricts sensitive classes to Australian regions and verifies provider processing locations. Supports PSPF information security and SA ISMF requirements. |
| Human oversight and transparency | Use-case register, risk tier and approvals align with the Voluntary AI Safety Standard guardrails. |
Mappings show how the design supports each control. They are not certifications, and EITSAA does not claim IRAP assessment or compliance for any reference architecture.
| Failure mode | Mitigation |
|---|---|
| A provider region is degraded or throttling | Circuit breaker, retry with backoff and fallback to a second approved provider for the same classification, otherwise queue or fail clearly. |
| Redaction misses an identifier | Second-pass response scanning, sampling reviews, provider-side guardrails and a documented incident path for suspected leakage. |
| Policy bug blocks a critical service | Policy tests in CI, canary rollout and an emergency override that is logged and time-limited. |
| Gateway outage | Multi-AZ, health checks and a documented manual degraded mode. Applications should handle model unavailability, not assume it. |
| Runaway spend from a loop | Per-key budgets with hard cut-off and alarms on token rate. |
Model tokens, provisioned throughput, gateway compute, GPU capacity for self-hosted models and log storage.
AWS · Kubernetes · vLLM
A private inference service running open-weight models with vLLM on EKS GPU nodes: queue-aware autoscaling, a governed model registry, weights loaded from encrypted S3 over a gateway endpoint, and a VPC with no route to the internet.
Agentic AI · MCP
A platform for agents that act in enterprise systems: a supervisor and specialist agents, an MCP gateway that holds every tool credential, a policy engine deciding each call, human approval for consequential steps, continuous evaluation and red-teaming, and a complete audit trail.
RAG · Government records
An assistant that answers staff questions from agency records. Retrieval enforces each user's access and classification before text reaches a model, answers carry citations to the authoritative record, and every interaction is logged so it can be managed under the State Records Act and produced under FOI.
Tell us your systems, constraints and timeline. We reply within one business day, and the initial conversation is free.