What to log
Log metadata and references by default, full content only where justified and masked.
Why. Balances investigation needs with privacy.
Trade-off. Less detail for some investigations.
AI governance and assurance · Observability · incident response
AI observability and incident response: tracing, quality metrics, drift detection, alerts and runbooks
How to see what AI systems are actually doing and respond when they misbehave: end-to-end tracing, quality and safety scoring on sampled traffic, drift and cost detection, alerting tied to runbooks and post-incident learning that feeds testing.
A reference architecture from EITSAA, not a named client system. Service limits, prices and product features change, so confirm current details before you commit to a design.
- NIST AI RMF (Measure and Manage)
- Voluntary AI Safety Standard
- ISO/IEC 42001
- APRA CPS 230 and CPS 234
- Notifiable Data Breaches scheme
- ACSC Essential Eight
Architecture
- AI and ML
- Messaging, events and workflow
- Monitoring, audit and analytics
- People and clients
- Security and identity
- Request or data flow
- Asynchronous or optional
- Boundary: region, account, network or trust zone
Request and data flow
Applications and agents emit OpenTelemetry spans for each model call, retrieval, tool use and guardrail decision, with trace identifiers.
The collector masks sensitive fields and forwards traces to a store where an investigator can follow a request end to end.
A sample of traffic is scored for quality, groundedness, safety and policy compliance, using automated judges and periodic human review.
Drift and cost detectors compare input topics, output patterns, refusal and block rates, latency and spend with baselines.
Alert rules set by risk tier fire when thresholds are breached, such as a quality drop, a spike in blocked injection attempts or abnormal spend.
The on-call engineer and an incident lead triage. Severity considers harm to people, data exposure, regulatory duties and impact on critical operations.
Runbooks guide containment: tighten a guardrail, roll back a version, disable a tool, switch to a fallback or trigger the kill switch. Notification duties are assessed.
A blameless post-incident review identifies root causes and actions, and adds new cases to evaluation sets.
The AI register and risk log are updated, and repeated issues may change a system's risk tier or controls.
Components and why they were chosen
| Component | What it does | Why this choice |
|---|---|---|
| Instrumentation (OpenTelemetry, OpenInference) | Standard spans for AI. | Portable across vendors. |
| Trace and analytics store (Langfuse, Azure Monitor, CloudWatch, Datadog) | Search and investigation. | Fast root-cause analysis. |
| Quality scoring jobs | Sampled evaluation in production. | Detects silent degradation. |
| Alerting and on-call (PagerDuty, Opsgenie, native tools) | Routing and escalation. | Right person, right time. |
| Runbook library | Tested response procedures. | Speed and consistency under pressure. |
| AI register | Link incidents to systems. | Learning feeds governance. |
Design decisions and trade-offs
Sampling
Score a sample and all flagged traffic.
Why. Controls cost.
Trade-off. Rare problems can be missed.
Alert design
Few, actionable alerts tied to runbooks.
Why. Avoids fatigue.
Trade-off. Needs tuning.
Incident classification
Define AI-specific incident types and severities.
Why. Consistent response and notification.
Trade-off. Must be maintained.
Notification duties
Integrate privacy, security and prudential notification checks into the runbook.
Why. Meets clocks under pressure.
Trade-off. Requires legal and risk input.
Well-architected pillars
Security
Masked telemetry, restricted access to traces and tamper-evident storage.
Reliability
Redundant collectors and alerting, with a secondary channel if the primary is down.
Performance
Asynchronous export so observability never slows requests.
Cost optimisation
Tune sampling and retention by tier.
Operational excellence
Regular game days for AI incidents and runbook updates.
Sustainability
Shorter retention for low-risk traces and compressed storage.
Network, identity and data
Networking
Collectors and stores are in a private network. Applications export through private endpoints. Alerting reaches on-call through approved channels.
Identity and least privilege
Access to traces is role-based and logged because traces can contain sensitive content. Incident roles are defined with deputies.
Data and encryption
Mask personal and confidential data before storage. Set retention by class. Preserve evidence under legal hold when an incident requires.
Security and compliance mapping
| Control | How this design addresses it |
|---|---|
| Monitoring and incident management | Continuous measurement and runbooks support Guidance for AI Adoption test and monitor, and NIST AI RMF Manage. |
| Event logging | Traces support investigation. Essential Eight. |
| Incident notification | Runbooks cover CPS 230 and CPS 234 timeframes for regulated entities and the Notifiable Data Breaches scheme. |
| Records | Incident records support assurance and ISO/IEC 42001 audits. |
| Continual improvement | Findings feed evaluation sets and the register. |
Mappings show how the design supports each control. They are not certifications, and EITSAA does not claim IRAP assessment or compliance for any reference architecture.
Failure modes, targets and scaling
- Target RTOUnder one hour for alerting and traces. Detection targets follow the tolerance level of the supported operation.
- Target RPOMinutes for trace data. Incident records are durable.
- Scaling and limitsSized by request volume, sampling rate and retention.
| Failure mode | Mitigation |
|---|---|
| Quality decays with no errors raised | Scored sampling and canary prompts with alerts. |
| Alert fatigue hides real issues | Prune alerts and tie each to a runbook. |
| Telemetry leaks personal data | Masking at the collector and access controls. |
| Observability platform is down during an incident | Redundancy and a minimal fallback view. |
| Lessons are not applied | Track actions to completion and add tests. |
Observability and runbooks
- Quality, groundedness and safety scores.
- Block, abstain and escalation rates.
- Latency, errors and spend.
- Mean time to detect and contain.
Cost drivers and levers
Trace storage, scoring tokens and alerting tools.
- Sample by tier.
- Short retention for low-risk traces.
- Aggregate before storing.
- Use small judges.
Implementation and fit
Phased approach
- Instrument applications and stand up masked tracing.
- Add scoring on sampled traffic and baseline dashboards.
- Add drift, cost and safety alerts with runbooks.
- Run game days and close the loop into evaluation and the register.
Anti-patterns to avoid
- Monitoring only uptime.
- Logging raw prompts with personal data.
- Alerts with no owner.
- Incident reviews that blame individuals.
- No fallback when the model is down.
When to use it
- Production AI used by customers, citizens or staff.
- Regulated entities with incident notification duties.
When not to
- Offline experiments with no production exposure.
More in ai governance and assurance
Data governance · privacy · lineage
Responsible AI data governance: classification, consent, lineage and retention for training, retrieval and prompts
How data is governed across the AI lifecycle: sources are classified and assessed for lawful use, consent and purpose are enforced, lineage tracks data into training sets, indexes and prompts, and retention and deletion apply to every derived copy.
AI governance · LLMOps
AI governance and LLMOps platform: model registry, prompt versioning, evaluations, guardrails and responsible AI approvals
The operating system for AI in an agency or enterprise: versioned models and prompts, automated evaluations and red-teaming as release gates, responsible AI approval with a risk register, and production monitoring of quality, drift, safety and cost.
Operating model · control framework
AI governance operating model and control framework: register, risk tiers, approvals, platform guardrails and assurance
How an organisation governs every kind of AI: leadership sets direction, an AI register and risk tiers decide how much review each system needs, an approval workflow gates release, platform guardrails enforce policy and assurance closes the loop with evidence.
Related insights
Discuss this architecture
Tell us your systems, constraints and timeline. We reply within one business day, and the initial conversation is free.