AI, LLM and real-time · AWS · Databricks · AgentCore
A reference architecture for an assistant that answers from governed company knowledge and takes careful actions: Databricks on AWS prepares and indexes the knowledge, a LangChain and LangGraph agent runs on Amazon Bedrock AgentCore with Bedrock models, and MCP tool microservices reach Microsoft 365, Xero, CRM, ServiceNow and government APIs through AgentCore Gateway and policy.
A reference architecture from EITSAA, not a named client system. Service limits, prices and product features change, so confirm current details before you commit to a design.
The user signs in with Amazon Cognito, or Cognito federated to Microsoft Entra ID, using MFA. The token is short-lived and carries the user's groups, which later decide what they may read and do.
The web app or Teams client sends the question with the token. CloudFront and AWS WAF filter abuse at the edge, and API Gateway (or the AgentCore Runtime JWT authoriser) checks the token, applies rate limits and passes only the validated identity on.
AgentCore Runtime starts or resumes the user's session in an isolated environment and runs the LangGraph agent. The graph has a fixed shape: classify, retrieve, answer, and only then consider tools, with a step and token budget.
The agent reads short-term conversation state and approved long-term facts from AgentCore Memory, keyed to that user. Memory never decides permissions, and long-term extraction is limited to fields the privacy assessment allows.
For knowledge questions the agent calls Databricks AI Search (formerly Vector Search) through the databricks-langchain retriever over PrivateLink, using a service principal. Hybrid keyword and vector search returns chunks, and a filter built from the user's groups is applied on every query, because index permissions alone do not trim results row by row.
The agent calls a Bedrock model through ChatBedrockConverse with the retrieved chunks marked as untrusted reference text. Bedrock Guardrails screen the input and output for denied topics, sensitive information and prompt attacks, and the answer cites its sources.
If the task needs an action or live data, the agent calls tools on AgentCore Gateway, which presents every approved tool as one MCP server. The LangChain MCP adapters load only the tools this agent is allowed, and Gateway semantic search keeps the list short.
Policy in AgentCore checks each call (user, agent, tool and arguments) with Cedar rules before it runs. For outbound calls, AgentCore Identity supplies a token scoped to the target, on behalf of the user where the service supports it, so no API key sits in the agent or the prompt.
Read-only calls proceed. Writes such as creating a ServiceNow ticket, sending an email through Microsoft Graph or raising a Xero draft pause: the LangGraph graph interrupts, shows the exact arguments to the user or an approver, and resumes only after approval.
Each MCP tool microservice (on AgentCore Runtime or Lambda) wraps one outside system with typed inputs, idempotency keys, timeouts and an egress allow-list. Results are filtered and size-limited before they return to the agent as untrusted data.
Ingestion runs separately: Lakeflow Connect or jobs pull documents from SharePoint, S3 and line-of-business systems, then parse, clean and chunk them, keeping source, owner, classification and access groups as columns on a Delta table in Unity Catalog.
A Delta Sync index in Databricks AI Search follows the chunk table, embedding new and changed rows automatically. Deletions and permission changes flow through the same table, so the index never holds content its source has removed.
| Component | What it does | Why this choice |
|---|---|---|
| Amazon CloudFront, AWS WAF and Amazon API Gateway | Edge delivery, bot and abuse filtering, throttling and token checks. | Standard AWS front door. Keeps the agent off the public internet. |
| Amazon Cognito or Microsoft Entra ID | User sign-in, MFA and group claims. | Use the identity platform staff already have. AgentCore accepts standard JWTs. |
| LangChain and LangGraph | Agent graph, retrieval chain, tool calling and human-in-the-loop interrupts. | Explicit, testable control flow, with integrations for Bedrock (langchain-aws), Databricks (databricks-langchain) and MCP (langchain-mcp-adapters). |
| Amazon Bedrock AgentCore Runtime | Serverless hosting for the agent and for MCP servers, with session isolation and JWT or IAM inbound auth. | Framework-agnostic, so the LangGraph code stays portable. |
| AgentCore Memory | Short-term session state and extracted long-term memory per user. | Managed memory with per-actor scoping, instead of a home-made store. |
| AgentCore Identity | Workload identities for agents and a token vault for outbound OAuth to third parties. | Agents never handle raw secrets, and user consent is recorded. |
| AgentCore Gateway and Policy in AgentCore | One MCP endpoint for all tools, built from OpenAPI, Smithy, Lambda or existing MCP servers, with Cedar policy on every call. | Central control of which agent can call what, enforced outside the model. |
| MCP tool microservices | Small servers, one per outside system: Microsoft Graph, Xero, CRM, ServiceNow and selected government APIs. | Small blast radius, separate owners and independent release and revocation. |
| Amazon Bedrock and Bedrock Guardrails | Model inference and content safety checks. | Australian cross-Region inference profiles can keep supported model traffic within Australian Regions. |
| Databricks on AWS: Lakeflow, Delta Lake and Unity Catalog | Ingestion, parsing, chunking, lineage and governance of the knowledge. | Use the platform where your data engineering already lives, with one catalogue for tables, files, models and indexes. |
| Databricks AI Search (formerly Vector Search) | Delta Sync vector index with hybrid search, filters and reranking. | The index updates from the governed table, so there is no second copy to keep in sync. |
| Amazon CloudWatch, AgentCore Observability and AWS CloudTrail | Traces, metrics, logs and API audit. | OpenTelemetry-compatible traces across agent, model and tool spans. |
| AWS KMS, AWS Secrets Manager and Amazon S3 | Customer-managed keys, secrets for ingestion jobs, and an Object Lock audit archive. | Encryption you control and tamper-evident evidence. |
Where retrieval lives
Why. The knowledge stays under Unity Catalog lineage and governance, next to the pipelines that produce it.
Trade-off. Two platforms to operate. If you have no Databricks estate, Bedrock Knowledge Bases or OpenSearch is simpler.
How document permissions are enforced
Why. Index access is granted to a service principal, and row or column level security is not applied to search results, so the application must trim them.
Trade-off. Group changes must flow through ingestion promptly, and the filter logic needs its own tests.
Where tools and credentials live
Why. A hijacked agent cannot use a key it never holds, and policy is deterministic and auditable.
Trade-off. More moving parts than calling APIs directly from the agent.
Which identity reaches the outside service
Why. The target system enforces the user's own entitlements and its audit log names the real person.
Trade-off. Consent flows and token refresh need design, and some services only offer app-level access.
Where embeddings and models run
Why. Data residency is a common requirement for Australian agencies and regulated businesses.
Trade-off. Some Databricks hosted models and features in Sydney rely on cross-geography routing, and Bedrock model availability differs by Region, so model choice may be narrower.
Deny-by-default tool policy, delegated or scoped tokens, approval for every write, Guardrails on input and output, permission filters on retrieval, PrivateLink and VPC endpoints, and egress allow-lists for tool services.
Managed runtimes across Availability Zones, idempotent tools with retries and timeouts, circuit breakers per outside service, LangGraph checkpoints, and a graceful answer when a tool or the index is unavailable.
Hybrid retrieval with reranking and a small top-k, streaming responses, parallel read-only tool calls, prompt caching where the model supports it, and semantic tool search so prompts list few tools.
Token use per conversation, agent loop length and index size drive cost. Budgets per session, model tiering and incremental Delta Sync keep it in check.
Versioned prompts, graphs, tools and Cedar policies, released through CI with evaluation gates, canaries and per-tool kill switches.
Use deterministic code for known steps, keep context small and avoid re-embedding unchanged content.
The agent and MCP tool services run in private subnets through AgentCore VPC mode, reaching AgentCore, Bedrock, S3 and Secrets Manager through interface and gateway VPC endpoints. The Databricks workspace sits in the same Region and is reached over AWS PrivateLink. Tool egress to Microsoft 365, Xero, CRM, ServiceNow and government APIs goes through a NAT gateway or proxy with a domain allow-list, and no tool server accepts traffic except from the Gateway.
Users sign in with Cognito or Entra ID and MFA. The token is validated at the edge and by AgentCore, then its claims drive both the retrieval filter and Cedar policy. Agents have workload identities, IAM roles are scoped per runtime and per tool, Databricks access uses a dedicated service principal with read access to the index only, and outbound OAuth tokens are issued per target with the narrowest scopes. Tokens are never passed straight through to a downstream API, in line with MCP security guidance.
Classify documents at ingestion, keep source, owner and access groups with every chunk, and exclude content that is not approved for AI use. Encrypt with customer-managed KMS keys on AWS and in Databricks where enabled. Set retention for Memory, traces and the audit archive, redact personal information from logs, and treat every retrieved chunk and tool result as untrusted data, never as instructions.
| Control | How this design addresses it |
|---|---|
| Restrict administrative privileges | Per-tool least privilege, no standing admin keys in agents and narrow OAuth scopes. Essential Eight. |
| Multi-factor authentication | MFA for users and approvers through Cognito or Entra ID. Essential Eight. |
| Human oversight | Approval before writes and external messages, and a kill switch per tool, support the Voluntary AI Safety Standard on human control. |
| Event logging and audit | AgentCore Observability traces, Policy decision logs, CloudTrail and an Object Lock S3 archive support ISM event logging. |
| Prompt injection and tool poisoning | Mapped to the OWASP LLM Top 10: untrusted-content handling, Guardrails prompt attack filters, reviewed tool descriptions pinned by version, and excessive agency limited by policy. |
| Data residency and cross-border disclosure | Sydney Region for AWS and Databricks, Australian inference profiles where supported, and documented exceptions support APP 8 assessments. |
Mappings show how the design supports each control. They are not certifications, and EITSAA does not claim IRAP assessment or compliance for any reference architecture.
| Failure mode | Mitigation |
|---|---|
| A document or email contains hidden instructions | Chunks and tool output are wrapped as untrusted, Guardrails screen for prompt attacks, policy decides by user and tool not by text, and writes need approval. |
| A tool description is changed to steer the agent (tool poisoning) | Tools are registered through Gateway from reviewed definitions, versions are pinned, changes need an owner's approval and evaluation, and unknown MCP servers are blocked. |
| A user sees content they should not | Group filters on every query, tests for the filter logic, prompt permission changes through ingestion, and sampling of answers against source permissions. |
| An outside service is slow or down | Timeouts, circuit breakers and a clear message to the user, while knowledge answers still work. |
| An action runs twice after a retry | Idempotency keys in each tool and checkpointed graph state. |
| Quality drops after a model, prompt or index change | Evaluation gate in CI, canary release and quick rollback of the version. |
Model tokens (agent loops and long context multiply them), AgentCore Runtime session time, Memory and Gateway usage, the AI Search endpoint and index size, ingestion compute, outside API limits, and evaluation runs with reviewer time.
If your documents and data already flow through Databricks, AI Search indexes the governed Delta tables directly, so lineage, ownership and deletions stay in Unity Catalog. If you have no Databricks estate, Bedrock Knowledge Bases or OpenSearch is usually simpler.
AWS lists AgentCore Runtime, Memory, Identity, Gateway, Policy and Observability in Sydney, and Databricks lists AI Search and Model Serving there. Some Databricks hosted models and features in Sydney rely on cross-geography routing, and Bedrock models differ by Region, so check the current regional availability pages and confirm where each model call is processed.
No. Outbound credentials come from AgentCore Identity at call time, scoped to one target and, where the service supports it, to the signed-in user. The agent code and prompts never see them.
Retrieved text and tool results are treated as data, Guardrails screen for prompt attacks, tools come only from reviewed and versioned Gateway targets, Cedar policy decides each call by user, tool and arguments, and any write needs human approval.
No. AgentCore Runtime is framework-agnostic and also runs Strands, CrewAI, LlamaIndex and custom code. We use LangGraph here for its explicit graph, checkpoints and human-in-the-loop interrupts.
An evaluation set built from real questions is scored for groundedness, correct citations, permission leaks and tool-use accuracy, with red-team cases for injection and data exfiltration. A failing run blocks release, and live traces are sampled for review.
AWS · AI, ML and deep learning
A plain explanation of how artificial intelligence, machine learning and deep learning relate, with a comparison table, practical guidance for South Australian businesses, councils and agencies, and three reference architectures on AWS: council document processing with managed AI services, demand forecasting with Amazon SageMaker AI, and road defect detection with deep learning and Amazon Bedrock.
AWS · chatbot · guardrails
A customer-facing chatbot with flow control and guardrails kept apart. A deterministic engine owns intents, slots, validation and retries. A catalog of controls across six layers (input, context, model, tool and action, output, human oversight) screens every message in and every reply out, and a small Bedrock model is called only between questions, all on pay-per-use serverless services.
AWS and Azure AI
A grounded question-answering system for internal knowledge: governed ingestion, hybrid retrieval with permissions, a guarded model call, citations, an audit log and a repeatable evaluation harness.
Tell us your systems, constraints and timeline. We reply within one business day, and the initial conversation is free.