Approach
RAG over fine-tuning for knowledge, with fine-tuning only for style or format.
Why. Knowledge changes and needs citations and deletions.
Trade-off. Quality depends on retrieval, so ingestion and evaluation need real engineering.
AI, LLM and real-time · AWS and Azure AI
RAG application with Amazon Bedrock or Azure OpenAI, a vector store, guardrails and evaluation
A grounded question-answering system for internal knowledge: governed ingestion, hybrid retrieval with permissions, a guarded model call, citations, an audit log and a repeatable evaluation harness.
A reference architecture from EITSAA, not a named client system. Service limits, prices and product features change, so confirm current details before you commit to a design.
- Australian AI Ethics Principles
- Voluntary AI Safety Standard
- Australian Privacy Principles (APP 11)
- ACSC Essential Eight
Architecture
- People and clients
- Network and edge
- AI and ML
- Security and identity
- Object and file storage
- Databases and data
- Monitoring, audit and analytics
- Request or data flow
- Asynchronous or optional
- Boundary: region, account, network or trust zone
Request and data flow
A signed-in user asks a question. The API authenticates the token and resolves the user's groups.
Retrieval runs a hybrid query (vector plus keyword), filtered by the document permissions captured at ingestion, then re-ranks the top results.
Input and output guardrails check for prompt injection, personal information, denied topics and unsafe content.
The model receives a prompt template with the retrieved passages and instructions to answer only from them and to cite sources, or to say it does not know.
The response, sources, model version, latency and guardrail outcomes are written to the audit store. Thumbs up or down feedback joins the same record.
Ingestion syncs documents from the source systems with their access control lists, detecting changes and deletions.
Documents are parsed, split into structure-aware chunks with metadata and embedded.
Chunks and vectors are upserted into the index. Deleted or restricted documents are removed, so permissions stay in step.
Components and why they were chosen
| Component | What it does | Why this choice |
|---|---|---|
| Amazon Bedrock or Azure OpenAI | Hosted foundation models with enterprise data terms and regional endpoints. | No model hosting to run. Data is not used to train the base models under the services' terms; verify the current terms and the region for inference. |
| Amazon Bedrock Guardrails or Azure AI Content Safety | Content filters, denied topics, PII redaction and prompt-attack detection. | Policy enforced outside the prompt, where it cannot be argued away. |
| pgvector on Aurora PostgreSQL, or Amazon OpenSearch Service | Vector similarity plus keyword search with metadata filters. | pgvector keeps vectors with relational data for moderate scale. OpenSearch suits large corpora and hybrid ranking. |
| Ingestion pipeline (Step Functions, Lambda or Azure Functions) | Sync, parse, chunk, embed and index. | Idempotent and replayable, so you can re-embed with a new model. |
| Document parsing (Textract or Document Intelligence) | OCR and layout for PDFs and scans. | Layout-aware parsing is the largest quality lever in practice. |
| RAG API (container or function) | Orchestration, permissions, prompt assembly and streaming. | Keeps logic testable and swappable across models. |
| Evaluation harness | Golden question set, retrieval metrics and answer grading. | Prevents silent regressions when prompts, models or chunking change. |
| Audit store (DynamoDB or SQL) | Immutable record of questions, sources and answers. | Supports review, complaints and improvement. |
Design decisions and trade-offs
Permissions
Store ACLs as metadata and filter at query time.
Why. The model must never see content the user could not open.
Trade-off. Permission syncing adds complexity and lag that must be monitored.
Retrieval
Hybrid search with a re-ranker, not vectors alone.
Why. Keyword matching catches identifiers and acronyms that embeddings blur.
Trade-off. More moving parts and latency.
Model choice
Abstraction layer with at least two candidate models.
Why. Models change quickly and pricing and quality vary by task.
Trade-off. Prompts need per-model tuning and tests.
Well-architected pillars
Security
Permission-aware retrieval, guardrails, private endpoints to the model, no secrets in prompts and prompt-injection testing.
Reliability
Timeouts, fallbacks to a second model, graceful degradation to search results only.
Performance
Streaming responses, caching of embeddings and common queries, limited context size.
Cost optimisation
Tokens dominate. Control context length, cache, and route simple questions to smaller models.
Operational excellence
Versioned prompts, index and model, shadow evaluation before change, feedback loop to content owners.
Sustainability
Smaller models where they pass evaluation and no unnecessary re-embedding.
Network, identity and data
Networking
Model access over VPC interface endpoints (PrivateLink for Bedrock) or Azure private endpoints. Vector store in private subnets. No public inference traffic from the application.
Identity and least privilege
User identity propagates to retrieval filters. The service uses a role with invoke permission for specific model IDs only. Separate roles for ingestion and for serving.
Data and encryption
Classify the corpus before indexing, and exclude categories that must not be searchable. Encrypt index and audit stores with KMS keys. Retain audit records per policy. Treat the prompt log as personal information where it contains it.
Security and compliance mapping
| Control | How this design addresses it |
|---|---|
| Privacy and AI governance | Use-case register, risk assessment, human oversight and transparency notices align with the AI Ethics Principles and the Voluntary AI Safety Standard. |
| Personal information security | Redaction, encryption and access controls support APP 11. |
| Restrict administrative privileges | Separate roles for model access, index admin and audit read. Essential Eight. |
| Logging | Immutable audit log supports investigation and complaints. |
Mappings show how the design supports each control. They are not certifications, and EITSAA does not claim IRAP assessment or compliance for any reference architecture.
Failure modes, targets and scaling
- Target RTOUnder an hour; index rebuilds from sources in hours
- Target RPOIndex is derived data, audit log near zero loss
- Scaling and limitsTens of millions of chunks are practical on OpenSearch; pgvector suits low millions per index. Model throughput quotas must be requested early.
| Failure mode | Mitigation |
|---|---|
| Hallucinated or unsupported answer | Grounding instruction, citation requirement, refusal path and evaluation with a no-answer test set. |
| Prompt injection from a document | Treat retrieved text as data, strip instructions, use guardrails and avoid tools with side effects in this tier. |
| Stale or leaked content after permission change | Frequent ACL sync, delete propagation and a tested kill switch to disable a source. |
| Model quota or outage | Fallback model, backoff and queue for non-interactive uses. |
Observability and runbooks
- Retrieval hit rate, answer groundedness score and refusal rate from the evaluation set run nightly.
- Latency per stage, token use and cost per question.
- User feedback dashboards and an owner for the top unanswered questions.
Cost drivers and levers
Input and output tokens, embedding generation, vector store capacity, document parsing.
- Trim context and re-rank to fewer chunks.
- Cache frequent answers.
- Use a smaller model for classification and routing.
- Embed once and re-embed only on change.
Implementation and fit
Phased approach
- Choose one narrow corpus and 50 real questions as the first evaluation set.
- Build ingestion with permissions and a basic retrieval baseline.
- Add guardrails, citations and the audit log, then pilot with a small group.
- Expand corpora and channels, guided by evaluation results.
Anti-patterns to avoid
- Indexing everything with no permissions model.
- No evaluation set, so quality is judged by demos.
- Letting the model call tools in the same flow without approval.
- Sending sensitive data to an endpoint outside Australia without review.
When to use it
- Policy, procedure, contract and support knowledge.
- Questions that need citations.
- Content that changes regularly.
When not to
- Decisions that affect people's rights without human review.
- Numerical reasoning over structured data (use text-to-SQL with controls or BI).
- Tiny corpora where plain search is enough.
More in ai, llm and real-time
AWS and Azure AI
AI agent with tool use and human approval
An agent that can do things, not just answer: a bounded planner, a registry of narrowly scoped tools, policy checks outside the model, human approval for consequential actions and a full action ledger.
WebRTC · LiveKit
Real-time video and voice for iOS and Android with LiveKit on AWS
A WebRTC platform for mobile apps built on the LiveKit SFU: token-based access, UDP and TURN/TLS connectivity, server-side recording to S3, webhooks, ingress for streams and scaling across nodes and regions.
WebRTC · Amazon Chime SDK
Meetings, voice and recording with the Amazon Chime SDK
A managed real-time platform on AWS: a Meetings API backend, mobile and web clients, media regions close to users, SIP media applications for phone calls and media pipelines for recording, all driven by Lambda and EventBridge.
Related insights
Discuss this architecture
Tell us your systems, constraints and timeline. We reply within one business day, and the initial conversation is free.