You cannot govern an LLM application on opinion. You need repeatable evidence that it works and that it resists misuse. Evaluation measures quality and safety on representative tasks. Red-teaming deliberately attacks the system to find weaknesses. Together they form the testing evidence that guidance such as the Guidance for AI Adoption, the NIST AI Risk Management Framework and ISO/IEC 42001 expect.
Build a golden set
A golden set is a curated collection of realistic inputs with expected behaviours. Draw from real, privacy-reviewed usage, subject matter experts and known edge cases. Include cases where the right answer is to refuse or ask for clarification. Version it, assign an owner and refresh it as use evolves. A set of a few hundred well-chosen cases is more valuable than thousands of generic ones.
Measure what matters
- Task quality: correctness, completeness and format.
- Groundedness: are claims supported by retrieved sources, and are citations accurate?
- Retrieval quality: does the right content come back, and only content the user may see?
- Safety: refusal behaviour, harmful content, privacy leakage.
- Fairness: consistent quality across relevant user groups and languages.
- Operational: latency and cost per task.
Automated judges can scale scoring, but calibrate them against human-labelled samples and re-check agreement periodically. A judge model should never be your only control.
Red-team methods
Use structured catalogues to plan coverage. The OWASP Top 10 for LLM Applications lists the main risk categories. MITRE ATLAS describes adversary tactics and techniques against AI systems. NIST’s Generative AI Profile lists risks specific to generative AI. Test for direct and indirect prompt injection, jailbreaks, system prompt extraction, data exfiltration through retrieval, tool abuse by agents, denial of wallet through excessive consumption, and harmful or biased outputs. Combine automated attack suites with creative human testers, including people who understand your domain.
Release gates and CI
Run fast smoke tests on every change and the full suite on release candidates. Define pass thresholds by risk tier and block promotion when they are not met. Record results with the exact model, prompt and guardrail versions so the evidence is reproducible. The AI evaluation and red-team pipeline architecture shows how this fits into delivery.
Keep testing after launch
Models, providers and users change. Schedule canary evaluations, sample production traffic for quality scoring and add every incident to the regression set. Repeat red-teaming after significant changes and at a regular cadence.
Report honestly
A good test report states what was tested, what was not, the results by category, residual risks and the decision taken. This is the document an approval board, auditor or buyer will read.
How EITSAA helps
We help you design and evidence AI governance that matches your risk, from the register and policy through to technical guardrails and audit-ready evidence. Explore AI Governance & Assurance, talk to our team or call 0402 401 584.
This is general guidance, not legal advice. Check current requirements with the relevant authority and take advice on your own position.