Sooner or later, an AI tool your organisation uses will get something wrong. A chatbot gives a resident the wrong rates due date, an assistant drafts a letter with a made-up policy reference, a staff member pastes client details into a public AI tool, or a vendor quietly changes the model behind a feature and answers start drifting. None of these needs to become a crisis if you have decided in advance what counts as an AI incident, who handles it and how you fix it. This guide sets out a practical AI incident response process for South Australian councils, agencies, not-for-profits and small businesses, and shows how it fits alongside your existing cyber and privacy plans.
This is general information, not legal advice. It reflects official guidance as at October 2026. Reporting duties depend on your sector, your contracts and where your users are, so confirm your own position with your legal adviser.
What counts as an AI incident
An AI incident is any event where an AI system causes, or nearly causes, harm or a breach of your own rules. Most are not cyber attacks. Typical examples:
- Wrong or invented answers that a customer, resident or staff member relied on, such as incorrect fees, eligibility, dates or policy advice.
- Privacy and confidentiality slips, such as personal information pasted into an unapproved tool, or an assistant revealing information a user should not see.
- Unfair or inconsistent outcomes, where an automated score, triage or recommendation treats some groups differently without a good reason.
- Misuse or manipulation, such as prompt injection that makes a chatbot ignore its instructions, or an agent taking an action it should not have been able to take.
- Undisclosed AI, where people were not told they were dealing with AI or that content was AI-generated. Our chatbot disclosure and AI content labelling guide covers what to tell people.
- Vendor and model changes, where a supplier updates a model, changes a setting or has an outage and quality drops without warning.
Count near misses too. A wrong answer caught by a staff member before it went out is the cheapest lesson you will get.
Why a separate AI playbook helps
- Cyber plans assume an attacker. Most AI incidents come from the system working as built but behaving badly, so the first steps are different: switch off a feature, correct information already given and check similar cases, rather than isolating devices.
- The harm is often spread out. One bad chatbot answer may have been given to hundreds of people before anyone notices, so you need logs to find who was affected.
- Existing duties still apply. If personal information is involved, the Notifiable Data Breaches scheme may apply (see our Privacy Act and notifiable data breaches guide). Misleading information given to customers can raise Australian Consumer Law issues, and agencies and councils have records and administrative law duties whether a decision was made by a person or with software.
- Buyers and frameworks expect it. The National AI Centre’s Guidance for AI Adoption and the earlier Voluntary AI Safety Standard both expect testing, monitoring and a way to act when things go wrong. Government buyers increasingly ask suppliers how they would detect and handle AI failures.
Keep it short and link it to your cyber incident response plan, so an event that is both, such as a prompt injection that leaks data, follows one joined-up process.
Severity levels that fit a small team
| Level | Example | Response |
|---|---|---|
| Low | An internal drafting assistant gives a poor summary that staff caught before use. | Log it, adjust the prompt or guidance, review in the monthly check. |
| Medium | A public chatbot gives a wrong but low-impact answer, such as opening hours, to a few people. | Fix within a business day, correct the content, check logs for others affected. |
| High | Wrong eligibility, payment or safety information given to the public, or personal information exposed. | Switch off or restrict the feature the same day, involve the accountable owner and privacy lead, contact affected people. |
| Critical | An agent takes harmful actions, widespread data exposure, or a risk to someone’s health or safety. | Stop the system immediately, run the cyber and privacy plans in parallel, brief executives and legal advisers. |
The response steps
- Detect and report. Give staff and the public an easy way to flag a bad answer, such as a feedback button in the chatbot and a single internal contact. Our AI failure modes guide lists what to monitor.
- Contain. Every AI feature should have an off switch or a safe fallback that someone on the team can use without a developer, such as hiding the chatbot and showing your phone number and contact form instead.
- Preserve evidence. Save the conversation logs, prompts, model and version, settings and any retrieved documents before they are overwritten. Without them you cannot tell how many people were affected.
- Assess the impact. Who received the output, what decisions they may have made, whether personal information was involved, and whether the same fault affects other cases.
- Fix the cause. Correct the source content, tighten guardrails, change permissions, add a test case that reproduces the fault, or roll back a vendor change. Our guardrails guide and red teaming and evaluation guide cover the technical side.
- Tell the right people. Correct the record with affected people in plain language, brief your accountable owner, and decide with your privacy lead and legal adviser whether any notification or report is required.
- Restart carefully and learn. Turn the feature back on only after the new test passes, then record what happened, what changed and who approved the restart in your AI register.
Who does what
- Accountable owner: the manager who owns the service the AI supports. Decides on switching off and restarting.
- Technical lead or provider: preserves logs, finds the cause and makes the fix.
- Privacy lead: assesses any personal information involved and runs the data breach process if needed.
- Communications or customer service: prepares the message to affected people and updates frontline staff.
In a small business one or two people may hold several of these roles. Write the names down anyway, with a backup for each. The AI governance framework guide explains how to record owners and risk tiers in an AI register.
Reporting duties to check
- Australia: there is no general AI incident reporting law. The Notifiable Data Breaches scheme, sector rules such as APRA CPS 230 for regulated financial entities, and your contracts may still require you to notify someone. Commonwealth agencies under the Digital Transformation Agency’s AI policy, and SA agencies under their own policies, should check their internal reporting requirements.
- European Union: under the EU AI Act, providers of high-risk AI systems must report serious incidents to market surveillance authorities, and providers of general-purpose AI models with systemic risk must report serious incidents to the AI Office. Following the Digital Omnibus, the Annex III high-risk rules apply from 2 December 2027. This mainly matters if you build or sell AI products used in the EU.
- Your suppliers: check whether your AI vendors must tell you about incidents, model changes and outages, and how quickly. If the contract is silent, ask. Our supplier due diligence guide includes questions to use.
Our global AI laws tracker covers the wider rules in Australia, the EU, US and UK.
A 30-day plan
- Week 1: list your AI tools and features, including those built into software you already use, and name an accountable owner for each.
- Week 2: write a one-page AI incident playbook with the severity levels, steps and contacts above, and link it to your cyber and privacy plans.
- Week 3: make sure every customer-facing AI feature has logging, a feedback route and an off switch someone can use without a developer.
- Week 4: run a 45-minute tabletop exercise, such as “our chatbot told 200 ratepayers the wrong due date”, and fix the gaps it shows.
Many AI incidents trace back to a design gap, such as a chatbot tool with too much access or a knowledge base anyone can edit. After an incident, or before launching a new AI feature, our guide to threat modelling apps and AI agents with STRIDE and MAESTRO helps you find those gaps layer by layer.
How EITSAA can help
If you are using chatbots, AI assistants or automated decisions and have no plan for when they go wrong, talk to us about an AI incident readiness review or call 0402 401 584. We can help you list your AI uses, write a short playbook, add logging, feedback and off switches to existing tools, and run a tabletop exercise with your team. Learn more about our AI governance and AI agents and chatbots services.