AI, LLM and real-time · AWS · Kubernetes · vLLM
A private inference service running open-weight models with vLLM on EKS GPU nodes: queue-aware autoscaling, a governed model registry, weights loaded from encrypted S3 over a gateway endpoint, and a VPC with no route to the internet.
A reference architecture from EITSAA, not a named client system. Service limits, prices and product features change, so confirm current details before you commit to a design.
Clients reach an internal Application Load Balancer from agency networks over Direct Connect or Transit Gateway. There is no public listener. TLS terminates at the load balancer, and mutual TLS can be added between the gateway and clients.
The load balancer forwards to the inference gateway, an OpenAI-compatible service that authenticates callers, applies per-team quotas and rejects oversized prompts before they consume GPU time.
The gateway routes to vLLM replicas using queue depth and KV-cache pressure, not round robin. vLLM uses continuous batching and paged attention to keep GPU utilisation high.
At pod start, weights are streamed from S3 through a gateway endpoint (no internet), cached on local NVMe and verified against the checksum recorded in the registry. Startup can take minutes for large models, so readiness gates wait for the load.
Promotion in the model registry copies an approved, scanned and evaluated model to the production weights bucket. Only the pipeline role can write there, and only approved versions are deployed.
KEDA scales replicas on queue depth or tokens per second, and Karpenter provisions or removes GPU nodes. A warm minimum of one replica avoids cold-start latency for interactive use.
vLLM and the NVIDIA DCGM exporter publish time to first token, tokens per second, GPU utilisation, memory and error metrics to Prometheus for dashboards and alerts.
Inference images are built from a pinned base, scanned, signed and pushed to ECR. The VPC reaches ECR through interface and S3 gateway endpoints, so nodes pull images without internet access.
| Component | What it does | Why this choice |
|---|---|---|
| Amazon EKS with managed GPU node groups or Karpenter (p4d, p5, g5, g6 families) | Runs GPU workloads with node auto-provisioning. | Kubernetes gives scheduling, rollout and autoscaling that bespoke EC2 scripts do not. |
| vLLM (alternatives: TGI, TensorRT-LLM, SGLang) | High-throughput serving with continuous batching and paged KV cache. | Strong throughput per GPU dollar and broad open-weight model support. |
| Inference gateway (Envoy AI Gateway, LiteLLM or a thin custom service) | OpenAI-compatible API, quotas and queue-aware routing. | Applications are portable between this tier and cloud providers. |
| Model registry (SageMaker Model Registry, MLflow or a Git-and-bucket convention) | Versions, model cards, licence, evaluation evidence and approval state. | Makes provenance and approval auditable. |
| S3 with SSE-KMS and a gateway endpoint | Weights storage reachable without the internet. | Durable, cheap and access-controlled. |
| KEDA and Karpenter | Event-driven pod scaling and just-in-time GPU node provisioning. | GPU capacity follows demand. |
| NVIDIA GPU Operator and DCGM exporter | Drivers, device plugin and GPU telemetry. | Standard, supportable GPU stack on Kubernetes. |
| ECR with image signing (cosign) and scanning | Controlled, scanned images. | Supply-chain integrity for the serving stack. |
Self-host or consume a managed model
Why. Managed services remove GPU operations and give access to frontier models.
Trade-off. You own capacity, patching, safety tuning and model quality.
Serving engine
Why. vLLM is easy to operate and fast enough for most workloads.
Trade-off. TensorRT-LLM can be faster on NVIDIA hardware but needs engine builds per model and GPU.
Kubernetes or managed endpoints
Why. Kubernetes pays off when you share GPUs and platform tooling.
Trade-off. Higher platform skill needs than a managed endpoint.
Parallelism and quantisation
Why. Fits bigger models on fewer GPUs and cuts cost.
Trade-off. Quantisation can degrade reasoning and multilingual quality, so run your evaluation set first.
Network isolation
Why. Removes exfiltration and dependency-pull risk from the serving plane.
Trade-off. Every dependency (images, weights, packages) must be mirrored internally.
Private subnets with no internet route, VPC endpoints with policies, KMS-encrypted weights, signed images, Pod Security Standards, network policies, IRSA roles per workload and no SSH to nodes.
Replicas across three AZs, PodDisruptionBudgets, readiness gates that wait for weights, and a warm minimum replica count. Capacity reservations for critical models.
Continuous batching, prefix caching, quantisation, right-sized GPU choice and queue-aware routing. Measure time to first token and tokens per second under realistic prompts.
GPU hours dominate. Use Savings Plans or capacity reservations for the baseline, Spot for batch jobs only, and scale to zero for non-production.
GitOps for manifests, canary rollouts of new model versions, load tests per release and runbooks for GPU node failures and out-of-memory events.
Right-size GPUs, quantise where safe, batch offline work into off-peak windows and shut down idle non-production pools.
Private subnets in three AZs with no NAT or internet gateway. Gateway endpoints for S3 and interface endpoints for ECR, STS, CloudWatch Logs, KMS and Secrets Manager. The internal ALB is reachable only from approved agency CIDR ranges. Security groups and Kubernetes NetworkPolicies allow only gateway-to-inference and inference-to-metrics traffic. Multi-GPU models stay within one node to avoid cross-node bandwidth limits.
Pods use IAM Roles for Service Accounts, with the inference role limited to read on one weights prefix. The pipeline role is the only writer. Cluster admin is just-in-time with MFA. Callers use workload identity at the gateway, and per-team API keys, where used, are rotated and scoped.
Prompts and outputs are transient in the serving pods and are not written to disk. Logs contain metadata only. Weights are licensed assets: record each model's licence terms in the registry and confirm they permit the intended government or commercial use before approval.
| Control | How this design addresses it |
|---|---|
| Application control | Only signed images from the internal ECR are admitted by policy (Kyverno or Gatekeeper). Essential Eight. |
| Patch applications and operating systems | Bottlerocket or AL2023 GPU AMIs refreshed on a schedule and rolled through node groups. Essential Eight. |
| Restrict administrative privileges | No node SSH, just-in-time cluster access and scoped IRSA roles. Essential Eight. |
| Network segmentation and no egress | Endpoint-only VPC supports ISM network and gateway guidance. |
| Data residency | All compute, storage and keys in ap-southeast-2. Supports PSPF and SA ISMF. |
Mappings show how the design supports each control. They are not certifications, and EITSAA does not claim IRAP assessment or compliance for any reference architecture.
| Failure mode | Mitigation |
|---|---|
| GPU node failure or Spot reclaim | Replicas in multiple AZs, PDBs and automatic node replacement. Do not run interactive models on Spot. |
| GPU capacity not available at scale-out | Capacity reservations or capacity blocks for the baseline, plus a fallback route to a managed provider if policy allows. |
| Out-of-memory from long contexts | Enforce maximum context at the gateway and tune vLLM memory utilisation and maximum sequences. |
| Bad model promotion | Evaluation gate in the registry, canary replica and one-step rollback to the previous version. |
| Weights bucket unavailable or corrupted | Versioned bucket, checksum verification and a cross-account copy of approved weights. |
GPU instance hours, idle warm capacity, S3 and cross-AZ data transfer, and engineering time.
Agentic AI · MCP
A platform for agents that act in enterprise systems: a supervisor and specialist agents, an MCP gateway that holds every tool credential, a policy engine deciding each call, human approval for consequential steps, continuous evaluation and red-teaming, and a complete audit trail.
RAG · Government records
An assistant that answers staff questions from agency records. Retrieval enforces each user's access and classification before text reaches a model, answers carry citations to the authoritative record, and every interaction is logged so it can be managed under the State Records Act and produced under FOI.
AWS · chatbot · guardrails
A customer-facing chatbot with flow control and guardrails kept apart. A deterministic engine owns intents, slots, validation and retries. A catalog of controls across six layers (input, context, model, tool and action, output, human oversight) screens every message in and every reply out, and a small Bedrock model is called only between questions, all on pay-per-use serverless services.
Tell us your systems, constraints and timeline. We reply within one business day, and the initial conversation is free.