Every AI system fails sometimes. Mature governance does not promise otherwise. It identifies how each system can fail, limits the damage, detects the failure quickly and recovers. This guide lists the failure modes we analyse most often and the design responses that work. Use it as the starting point for a failure mode and effects analysis (FMEA) for each system in your register.
The main failure modes
| Failure mode | What it looks like | Design response |
|---|---|---|
| Hallucination | Fluent, confident, wrong output | Grounding in retrieved sources, citations, abstain when unsure, human review for consequential use |
| Bias and unequal performance | Worse accuracy or outcomes for some groups | Representative test sets, fairness metrics by group, review of features and proxies |
| Data and concept drift | Accuracy decays as the world changes | Drift monitoring, scheduled revalidation, retraining governance |
| Data leakage | Sensitive data exposed in outputs, logs or to providers | Access-controlled retrieval, PII redaction, provider terms, log masking |
| Prompt injection | Hidden instructions in content hijack behaviour | Treat content as data, isolate tools, validate outputs, red-team |
| Tool misuse and excessive agency | An agent takes an unintended or harmful action | Least privilege, approval gates, limits, reversible actions, kill switch |
| Over-reliance | People stop checking and accept outputs | Interface design that shows confidence and sources, sampling audits, training |
| Model deprecation | A provider retires or changes a model | Version pinning, regression suites, tested alternates, contract notice terms |
| Vendor outage | Provider or region unavailable | Multi-region or multi-provider fallback, graceful degradation, manual process |
| Silent degradation | Quality falls without any error | Quality scoring on sampled traffic, canary tests, alerting on scores |
| Runaway cost | Loops or abuse drive spend | Token, step and spend limits, budgets and alerts |
Design principles
- Limit blast radius. Constrain what a failure can touch with scoped permissions, small actions and reversibility.
- Fail safe. When a check fails or a model is unavailable, the system should hand over to a person or a simpler process, not guess.
- Detect, don’t assume. Silent failures need active measurement because nothing will crash.
- Keep humans meaningfully in the loop. Review only works when reviewers have time, context and authority to say no.
- Plan recovery. Define recovery objectives and rehearse the runbook for the AI-specific incidents most likely to occur.
Connect AI failures to resilience obligations
When an AI system supports a critical operation, its failure modes belong in your CPS 230 resilience planning: tolerance levels, fallback procedures and service provider exit plans for model providers. See provider due diligence and exit planning.
Make it a living analysis
Revisit the analysis after incidents, model changes and new features. Feed findings into evaluation sets so the same failure is tested forever. The AI observability and incident response architecture shows how detection and runbooks fit together.
How EITSAA helps
We help you design and evidence AI governance that matches your risk, from the register and policy through to technical guardrails and audit-ready evidence. Explore AI Governance & Assurance, talk to our team or call 0402 401 584.
This is general guidance, not legal advice. Check current requirements with the relevant authority and take advice on your own position.