Where the model runs
Vertex AI managed endpoints, dedicated endpoints for open models when needed.
Why. No GPU operations for most use cases.
Trade-off. Model and feature availability differs by region, and you must check residency per model.
AI, LLM and real-time · Google Cloud
LLM inference on Google Cloud with Vertex AI, Cloud Run and VPC Service Controls
A private, governed Gemini and Model Garden inference platform: Cloud Armor at the edge, Cloud Run for the application, Vertex AI over Private Service Connect, Pub/Sub for async jobs and BigQuery for audit and cost analytics.
A reference architecture from EITSAA, not a named client system. Service limits, prices and product features change, so confirm current details before you commit to a design.
- Australian Privacy Principles (APP 11)
- ACSC Essential Eight
- Google Cloud Sydney (australia-southeast1) and Melbourne (australia-southeast2)
Architecture
- People and clients
- Network and edge
- Compute and containers
- AI and ML
- Messaging, events and workflow
- Databases and data
- Security and identity
- Monitoring, audit and analytics
- Request or data flow
- Asynchronous or optional
- Boundary: region, account, network or trust zone
Request and data flow
Users reach a global external Application Load Balancer. Cloud Armor applies WAF rules, rate limits and optional geo controls, and managed certificates terminate TLS.
A serverless network endpoint group routes to Cloud Run. Identity-Aware Proxy or application-level OIDC authenticates users, and ingress is set to internal and load-balancer only.
The service authenticates the user, applies quotas and retrieves context from Vertex AI Search, AlloyDB with pgvector or another store, using the caller's permissions.
Calls to Vertex AI go through Private Service Connect for Google APIs so traffic stays on Google's private network, from a service account with only the aiplatform.user role.
Vertex AI serves Gemini or a Model Garden model from a regional endpoint. The VPC Service Controls perimeter blocks data exfiltration to projects outside the perimeter.
Long-running or batch work is published to Pub/Sub and processed by Cloud Run jobs or workers, with dead-letter topics for failures.
Request metadata, token counts, latency, safety ratings and evaluation scores stream to BigQuery for audit, cost allocation and quality monitoring. Prompts and responses are logged only where policy permits.
Components and why they were chosen
| Component | What it does | Why this choice |
|---|---|---|
| Cloud Armor and external Application Load Balancer | WAF, DDoS protection, rate limiting and TLS. | Protects a public endpoint before any compute is invoked. |
| Cloud Run | Containerised gateway and application, scaling to zero. | Low operations, per-request scaling and fine-grained IAM invoker control. |
| Vertex AI (Gemini) | Managed foundation models with safety settings, grounding and function calling. | Enterprise controls and regional endpoints. Confirm model availability and data-processing location for your chosen region. |
| Vertex AI Model Garden | Open and partner models you can deploy to dedicated endpoints. | Use when you need a specific open model or dedicated capacity. |
| Private Service Connect | Private endpoints for Google APIs. | Keeps API traffic off the public internet and works with on-premises connectivity. |
| VPC Service Controls | Perimeter around data and AI services. | Strong exfiltration control, though it needs careful rule design for ingress and egress. |
| Pub/Sub | Async messaging with retries and DLQs. | Decouples slow generations from user requests. |
| BigQuery | Analytics on usage, cost and quality. | Serverless SQL for dashboards and audits. |
| Cloud KMS (CMEK) and Secret Manager | Customer-managed keys and secrets. | Control and audit of encryption. |
Design decisions and trade-offs
Connectivity
Private Service Connect plus VPC Service Controls.
Why. Private path and exfiltration protection.
Trade-off. Perimeter rules take effort and can block legitimate integrations until tuned.
Application platform
Cloud Run over GKE.
Why. Less to run, fast scaling.
Trade-off. Less control for long-lived connections or custom networking, where GKE fits better.
Gateway
A thin LLM gateway layer (quotas, redaction, routing) in Cloud Run or Apigee.
Why. One control point for cost and policy across models.
Trade-off. Another component to maintain, and it must be fast.
Logging prompts
Log metadata by default and prompts only with a purpose and retention.
Why. Prompts hold personal information.
Trade-off. Harder debugging without sample capture, so use consented samples.
Well-architected pillars
Security
Cloud Armor, IAM least privilege with service accounts, VPC-SC perimeter, CMEK, Secret Manager, and Model Armor or safety filters for prompt and response screening.
Reliability
Regional services with retries and fallbacks to a second model or region, Pub/Sub DLQs, and quota monitoring.
Performance
Streaming responses, context caching where supported, regional endpoints near users, and concurrency tuned on Cloud Run.
Cost optimisation
Tokens per request dominate. Control them with quotas, caching, routing to smaller models and BigQuery cost attribution.
Operational excellence
Terraform, Cloud Build or GitHub Actions with workload identity federation, SLOs in Cloud Monitoring.
Sustainability
Serverless scale to zero and right-sized models.
Network, identity and data
Networking
Cloud Run with ingress restricted to the load balancer and egress through a Serverless VPC connector or direct VPC egress into a VPC with Private Google Access or PSC. No public IPs on workloads. Hybrid connectivity by Cloud Interconnect or VPN if needed.
Identity and least privilege
Workforce identity via Cloud Identity or federation from Entra ID. Service accounts per service with minimum roles. No service account keys, using workload identity federation for CI. IAM Conditions for environment scoping.
Data and encryption
Classify data before sending to a model. Use CMEK where policy requires, keep processing in Australian regions where models support it, and confirm Google's data-use terms for the product. Sensitive Data Protection (DLP) can redact before calls.
Security and compliance mapping
| Control | How this design addresses it |
|---|---|
| Restrict administrative privileges | Role design and no key files. Essential Eight. |
| Multi-factor authentication | Enforced at the identity provider. Essential Eight. |
| Data exfiltration control | VPC-SC perimeter supports ISM guidance on boundary protection. |
| Privacy | Redaction, regional processing and retention support APP 8 and APP 11. |
Mappings show how the design supports each control. They are not certifications, and EITSAA does not claim IRAP assessment or compliance for any reference architecture.
Failure modes, targets and scaling
- Target RTOUnder an hour from code
- Target RPOBigQuery and stores per their own backups. Inference is stateless
- Scaling and limitsGoverned by Vertex AI quotas per model and region, which you request in advance.
| Failure mode | Mitigation |
|---|---|
| Model quota or regional capacity limit | Quota alerts, retry with backoff and fallback model or region. |
| Prompt injection through retrieved content | Model Armor or safety filters, content isolation and tool restrictions. |
| Perimeter blocks a legitimate flow | Dry-run mode, VPC-SC logs and staged rollout. |
| Runaway cost from a loop | Per-user and per-key quotas and budget alerts. |
Observability and runbooks
- Cloud Monitoring SLOs for latency and error rate.
- BigQuery dashboards for tokens, cost and safety blocks.
- Evaluation runs with Vertex AI evaluation or your own golden set on every prompt change.
Cost drivers and levers
Input and output tokens, dedicated endpoint hours, retrieval stores and egress.
- Route by task to smaller Gemini models.
- Cache context and answers.
- Limit output tokens.
- Batch non-urgent work.
Implementation and fit
Phased approach
- Landing zone with perimeter in dry-run and a sandbox project.
- Gateway on Cloud Run with one model and logging.
- Add retrieval, guardrails and evaluation.
- Enforce the perimeter and roll out to users.
Anti-patterns to avoid
- Calling Vertex from the browser.
- Service account keys in CI.
- One shared service account.
- No budget alerts.
When to use it
- Organisations standardised on Google Cloud or Workspace.
- Apps needing long-context or multimodal Gemini models.
When not to
- Teams with no Google Cloud skills and a Microsoft-only estate.
- Where model availability in Australian regions does not meet policy.
More in ai, llm and real-time
Azure
LLM inference on Azure with Azure OpenAI, an APIM AI gateway and AI Search
A governed Azure OpenAI platform: Front Door and API Management as the AI gateway, private endpoints, Entra ID managed identities, AI Search for retrieval, Content Safety, Cosmos DB for conversation state and Monitor for cost and quality.
AI platform
AI platform for LLM bots and agents integrated with business processes
A reusable platform, not a single bot: channels, an agent runtime, a model gateway, retrieval, a tool registry with policy, a durable workflow engine, human approvals, CRM and ERP connectors, evaluation and cost control.
AI gateway · multi-cloud
Sovereign LLM gateway for government: one policy point across Bedrock, Azure OpenAI, Vertex AI and self-hosted models
A central AI gateway that every agency application calls instead of a model provider. It authenticates the caller, applies policy and PII redaction, routes each request to an approved Australian-region model, enforces budgets and writes a tamper-evident audit record.
Related insights
Discuss this architecture
Tell us your systems, constraints and timeline. We reply within one business day, and the initial conversation is free.