AI, LLM and real-time · Azure
A governed Azure OpenAI platform: Front Door and API Management as the AI gateway, private endpoints, Entra ID managed identities, AI Search for retrieval, Content Safety, Cosmos DB for conversation state and Monitor for cost and quality.
A reference architecture from EITSAA, not a named client system. Service limits, prices and product features change, so confirm current details before you commit to a design.
Users and apps reach Azure Front Door with WAF. Premium tier lets Front Door use Private Link to the origin.
API Management is the AI gateway. It validates the Entra ID token, applies per-product token quotas and rate limits, and can load balance across multiple model deployments or regions with circuit breaking.
The orchestrator receives the validated request with the caller's identity and groups.
It queries Azure AI Search with a hybrid (keyword and vector) query plus semantic ranking, applying security trimming filters from the caller's groups.
Prompts and retrieved content are checked by Azure AI Content Safety, including Prompt Shields for injection, and outputs are screened for harmful content.
The orchestrator calls the Azure OpenAI deployment using its managed identity over a private endpoint, with key-based authentication disabled.
Conversation state and feedback go to Cosmos DB with a time-to-live that matches the retention policy.
API Management emits token usage metrics and the gateway logs to Application Insights, which feed cost allocation by team and product.
| Component | What it does | Why this choice |
|---|---|---|
| Azure Front Door Premium with WAF | Global entry, WAF and Private Link origins. | Keeps APIM and apps private. |
| Azure API Management (AI gateway) | Token limits, semantic caching, backend pools and metrics for Azure OpenAI and other model APIs. | A single control point for quotas, chargeback and resilience across model deployments. |
| Azure OpenAI in Foundry Models | GPT family and embedding model deployments with regional, data-zone or global deployment types and provisioned throughput options. | Choose deployment type by residency and performance needs. Check model availability in Australia East. |
| Azure AI Search | Hybrid retrieval, semantic ranking and security trimming. | Mature enterprise retrieval with built-in indexers and integrated vectorisation. |
| Azure AI Content Safety | Harm categories, Prompt Shields and groundedness detection. | Policy enforcement independent of the model. |
| Azure Cosmos DB | Low-latency conversation and session store. | Global distribution is available and TTL gives retention control. |
| Container Apps | Orchestrator and ingestion workers. | Managed containers inside the VNet. |
| Microsoft Entra ID and managed identities | User and workload identity. | No keys to rotate. |
| Key Vault, Monitor and Defender for Cloud | Secrets, telemetry and posture. | Native governance. |
Deployment type
Why. Aligns data-processing location, latency and cost.
Trade-off. Regional capacity and model versions can lag, so confirm availability early.
AI gateway
Why. Cost control, chargeback and resilience.
Trade-off. Extra hop and APIM tier cost.
Authentication to the model
Why. Removes shared keys.
Trade-off. Local tools need developer RBAC roles.
Retrieval
Why. The model never sees content the user cannot access.
Trade-off. Group and ACL syncing in the index.
Responsible AI
Why. Measured, repeatable safety.
Trade-off. Added latency and cost per call.
Private endpoints, Entra ID, managed identities, Key Vault CMK, WAF, Prompt Shields and security-trimmed retrieval.
Zone-redundant services, APIM backend pools across deployments with circuit breakers, and queued retries for batch work.
Provisioned throughput for predictable latency, semantic caching in APIM, streaming responses and right-sized chunking.
Tokens and provisioned units. APIM token metrics enable per-team budgets and alerts.
Bicep or Terraform, GitHub Actions with federated credentials, evaluation gates and Azure Monitor workbooks.
Smaller models for routing and extraction, caching and scheduling batch work off peak.
Spoke VNet with private endpoints for Azure OpenAI, AI Search, Cosmos DB, Storage and Key Vault. Public network access disabled on those services. Private DNS zones for privatelink names. Egress via the hub firewall where required.
Users sign in with Entra ID and Conditional Access. Application roles map to APIM products. The orchestrator uses a managed identity with Cognitive Services OpenAI User and Search Index Data Reader roles only. PIM for administrators.
Azure OpenAI does not use your prompts or completions to train base models under the service terms, but confirm current terms and the data-processing location of your deployment type. Encrypt with customer-managed keys where required, set TTL on stored conversations, and redact personal information before logging.
| Control | How this design addresses it |
|---|---|
| Multi-factor authentication | Conditional Access. Essential Eight. |
| Restrict administrative privileges | PIM and role-scoped managed identities. Essential Eight. |
| Responsible AI | Content Safety, evaluation and human oversight align with the Voluntary AI Safety Standard. |
| Privacy | Private networking, retention and logging controls support APP 11. |
Mappings show how the design supports each control. They are not certifications, and EITSAA does not claim IRAP assessment or compliance for any reference architecture.
| Failure mode | Mitigation |
|---|---|
| Model deployment throttled (429) | APIM retries and load balancing across deployments, with fallback to another region if residency allows. |
| Prompt injection through a document | Prompt Shields, data isolation and no side-effect tools in this tier. |
| Index permissions stale | Frequent ACL sync and a source kill switch. |
| Private DNS misconfiguration | DNS in code and connectivity tests in the pipeline. |
Tokens or provisioned throughput units, AI Search units, APIM tier and Front Door.
AI platform
A reusable platform, not a single bot: channels, an agent runtime, a model gateway, retrieval, a tool registry with policy, a durable workflow engine, human approvals, CRM and ERP connectors, evaluation and cost control.
AI gateway · multi-cloud
A central AI gateway that every agency application calls instead of a model provider. It authenticates the caller, applies policy and PII redaction, routes each request to an approved Australian-region model, enforces budgets and writes a tamper-evident audit record.
AWS · Kubernetes · vLLM
A private inference service running open-weight models with vLLM on EKS GPU nodes: queue-aware autoscaling, a governed model registry, weights loaded from encrypted S3 over a gateway endpoint, and a VPC with no route to the internet.
Tell us your systems, constraints and timeline. We reply within one business day, and the initial conversation is free.