AI, LLM and real-time · AWS · chatbot · guardrails
A customer-facing chatbot with flow control and guardrails kept apart. A deterministic engine owns intents, slots, validation and retries. A catalog of controls across six layers (input, context, model, tool and action, output, human oversight) screens every message in and every reply out, and a small Bedrock model is called only between questions, all on pay-per-use serverless services.
A reference architecture from EITSAA, not a named client system. Service limits, prices and product features change, so confirm current details before you commit to a design.
The visitor opens the chat widget, which is loaded only on click so it adds nothing to the page until it is used, and types a message or taps a suggested answer.
The widget sends a small JSON task over HTTPS. AWS WAF applies a rate-based rule and managed rule groups before any compute runs.
Allowed requests reach API Gateway, which enforces CORS to the site origin, a maximum body size and per-route throttling.
The function runs the input guardrails first: profanity with obfuscation handling, prompt-injection patterns, topic scope, length and personal data masking. A blocked message never reaches the model.
The cleaned message goes to the conversation engine, which detects the intent, fills slots, validates each answer and decides the next question.
Slots, the retry counter and the phase are stored per session with a short time to live, so the conversation resumes after a dropped connection and is deleted soon after.
Between specific questions the engine calls a small Bedrock model through the Converse API to rephrase the next question, read a free-text answer into a slot or write the closing summary. The system prompt is fixed on the server and token limits are small.
The model reply is returned to the function, never straight to the visitor.
Output moderation checks the reply for links, prices, promises, personal data, leaked instructions and unsafe language. Anything that fails is replaced by scripted wording.
When the visitor confirms, asks for a person or reaches the retry limit, the engine hands over to the contact form or a staff queue with the confirmed details.
Masked counts of intents, retries, guardrail hits, token use and model latency go to metrics and alarms. Raw visitor text is not stored.
| Component | What it does | Why this choice |
|---|---|---|
| AWS WAF (rate-based rule, AWS managed rule groups) | Filters abusive traffic before it costs anything. | Cheapest place to stop floods and known bad requests. |
| Amazon API Gateway HTTP API | Public endpoint with CORS, throttling and a small payload limit. | Pay per request with built-in throttling. |
| AWS Lambda (Node.js, arm64) | Runs guardrails, the conversation engine and model calls. | No idle cost and scales to zero. |
| Amazon DynamoDB (on-demand, TTL) | Session slots and retry counters. | Pay per request with automatic expiry. |
| Amazon Bedrock Converse API (small model) | Rephrasing, interpretation and summaries. | Pay per token, model swappable by configuration. |
| Amazon Bedrock Guardrails (optional) | Managed content, topic and PII filters as a second layer. | Defence in depth that policy owners can tune. |
| Amazon CloudWatch and AWS Budgets | Metrics, alarms and spend alerts. | Early warning on abuse and cost. |
Flow control and guardrails
Why. Each can be tested, owned and evidenced separately, and guardrails apply to every flow.
Trade-off. Two things to maintain, mapped to each other by identifiers.
Who owns the conversation
Why. Behaviour is predictable, testable and cheap, and a model failure cannot derail the journey.
Trade-off. Less open-ended than a model-driven chat.
Where the model is called
Why. Keeps token use small and each call easy to validate.
Trade-off. More orchestration code than one large prompt.
Guardrail placement
Why. Client checks improve experience, server checks are the control.
Trade-off. Rules must stay in sync, so share one code module.
What happens when the model is unavailable
Why. Availability does not depend on a model.
Trade-off. Wording is less natural during an outage.
Model size
Why. Tasks are narrow and spend stays predictable.
Trade-off. Not suited to open-domain questions.
Input and output guardrails, personal data masked before any model call, fixed server-side system prompt, CORS limited to the site origin and least-privilege IAM for model invocation only.
Scripted fallback for every model step, retries with a limit and a handoff path that does not depend on AI.
Small prompts and token caps keep replies fast, and the widget code is loaded on demand.
Pay per request and per token with throttles, daily ceilings and budget alarms.
Metrics for retries, guardrail hits and latency, plus regular replay of test conversations.
Scale-to-zero compute and short, capped model calls.
The endpoint is public by design, so it sits behind WAF and API Gateway throttling. The function reaches Bedrock and DynamoDB over AWS endpoints, and no inbound path to the data stores exists.
Visitors are anonymous and receive a random session id. The function role can only invoke the chosen model, read and write its session table and write logs. Administrators use SSO with MFA.
Visitors are asked for no personal details. Anything that looks like personal data is masked before storage or model calls. Sessions expire quickly, logs hold counts rather than text, and data stays in the Sydney region.
| Control | How this design addresses it |
|---|---|
| Input layer: limits, toxicity, injection, prompt extraction, sensitive data, scope, unsafe advice (IN-01 to IN-08) | Address OWASP LLM01, LLM02, LLM06, LLM07, LLM09 and LLM10, and VAISS guardrails 2, 3 and 4. |
| Context layer: allow-listed context and data minimisation (CX-01, CX-02) | Only validated answers and the scripted question reach the model, and visitor text is delimited data. OWASP LLM01, LLM02 and LLM08, VAISS 3. |
| Model layer: fixed server prompt, token caps, confidence threshold, retry, fallback model, evaluation set (MD-01 to MD-05) | OWASP LLM01, LLM07, LLM09 and LLM10, VAISS 2 and 4, NIST Measure and Manage. |
| Tool and action layer: action allow-list and least privilege (TL-01, TL-02) | The only action is a hand-off to a fixed address, and the model has no tools. OWASP LLM06, VAISS 5. |
| Output layer: moderation, schema validation, leakage and grounding checks (OT-01 to OT-04) | OWASP LLM05, LLM07 and LLM09, VAISS 4 and 6. |
| Human oversight: crisis escalation, retry limit, audit log, disclosure (OV-01 to OV-04) | VAISS 5, 6, 7 and 9, NIST Govern and Manage, ISO/IEC 42001 records and oversight expectations. |
Mappings show how the design supports each control. They are not certifications, and EITSAA does not claim IRAP assessment or compliance for any reference architecture.
| Failure mode | Mitigation |
|---|---|
| Hallucination: a reply states something that was not supplied | Grounding check against allowed facts (OT-04), confidence threshold (MD-03) and scripted fallback. |
| Over-reliance: a person is left with automation where a decision needs a human | Retry limit and hand-off (OV-02), disclosure that the assistant is automated (OV-04) and crisis escalation (OV-01). |
| Visitor tries to override the instructions | Reject at the input guardrail, count it as a retry and offer a person after the limit. |
| Model returns a link, price or promise | Output moderation replaces it with scripted wording and logs the event. |
| Model is slow or unavailable | Short timeout, one retry, then scripted wording for the rest of the step. |
| Traffic spike or scripted abuse | WAF rate rule, API throttle, per-address limit and a daily ceiling. |
| Visitor types personal details | Mask before use, never store raw text and remind them to use the contact form. |
Requests, model tokens per conversation and logging volume.
AWS and Azure AI
A grounded question-answering system for internal knowledge: governed ingestion, hybrid retrieval with permissions, a guarded model call, citations, an audit log and a repeatable evaluation harness.
AWS and Azure AI
An agent that can do things, not just answer: a bounded planner, a registry of narrowly scoped tools, policy checks outside the model, human approval for consequential actions and a full action ledger.
WebRTC · LiveKit
A WebRTC platform for mobile apps built on the LiveKit SFU: token-based access, UDP and TURN/TLS connectivity, server-side recording to S3, webhooks, ingress for streams and scaling across nodes and regions.
Tell us your systems, constraints and timeline. We reply within one business day, and the initial conversation is free.