Judge models
Use judges with calibrated rubrics and periodic human agreement checks.
Why. Scales scoring.
Trade-off. Judges have bias, so they are not the sole control.
AI governance and assurance · Evaluation · red-teaming · CI
AI evaluation and red-team pipeline: golden sets, adversarial tests, release gates and CI
A pipeline that turns testing into a release gate: golden sets and adversarial suites run in CI against every change, scores are compared with thresholds by risk tier, failures block promotion and results become evidence.
A reference architecture from EITSAA, not a named client system. Service limits, prices and product features change, so confirm current details before you commit to a design.
- OWASP Top 10 for LLM Applications
- MITRE ATLAS
- NIST AI RMF Generative AI Profile
- Voluntary AI Safety Standard (testing)
- ISO/IEC 42001
- ACSC Essential Eight
Architecture
- Delivery tooling
- AI and ML
- Security and identity
- Monitoring, audit and analytics
- Network and edge
- Request or data flow
- Asynchronous or optional
- Boundary: region, account, network or trust zone
Request and data flow
A change to a prompt, model version, retrieval setting, guardrail or code opens a pull request, which triggers the pipeline.
The runner builds the candidate and executes the golden sets: curated, versioned cases for correctness, groundedness, refusal and format.
Adversarial suites attack the candidate with prompt injection, jailbreaks, data-exfiltration probes, tool-abuse scenarios and consumption attacks, drawn from OWASP and MITRE ATLAS techniques.
Responses are scored by deterministic checks and by judge models that are calibrated against human-labelled samples.
Human red teamers run exploratory sessions for new attack ideas and domain-specific harms. Their findings are turned into tests.
Scores are compared with thresholds set by risk tier and with the previous release to detect regressions.
Results, versions and decisions are written to the evidence store so the release can be reproduced and audited.
If thresholds pass, the signed candidate is promoted to the production gateway. If not, promotion is blocked and the failure is reported.
Production incidents and user feedback become new golden and adversarial cases, so the same failure is tested forever.
Components and why they were chosen
| Component | What it does | Why this choice |
|---|---|---|
| CI platform (GitHub Actions, GitLab CI, Azure DevOps) | Runs on every change. | Testing integrated into delivery. |
| Evaluation frameworks (promptfoo, Inspect, Ragas, Bedrock Evaluations, Azure AI Evaluation) | Quality and safety metrics. | Repeatable scoring. |
| Adversarial tooling (garak, PyRIT and custom suites) | Automated attack generation. | Breadth of attack coverage. |
| Golden set repository | Versioned, owned test cases. | Tests are assets. |
| Release gate policy | Thresholds as code. | Consistent, auditable decisions. |
| Evidence store | Results and versions. | Audit and reproducibility. |
Design decisions and trade-offs
Suite depth
Fast smoke suite on each commit, full suite on release candidates.
Why. Balances speed and coverage.
Trade-off. Some issues surface later.
Thresholds
Set by risk tier and tighten over time.
Why. Proportionate rigour.
Trade-off. Needs governance for exceptions.
Test data
Synthetic or de-identified data unless authorised.
Why. Protects privacy.
Trade-off. May miss real-world edge cases.
Human red team
Regular sessions, plus after major changes.
Why. Humans find novel attacks.
Trade-off. Cost and scheduling.
Well-architected pillars
Security
Isolated evaluation environments with no route to production data, protected attack corpora and signed promotion.
Reliability
Deterministic seeds where possible and rerun of flaky tests with variance tracking.
Performance
Parallel suites and caching of unchanged results.
Cost optimisation
Evaluation tokens cost money. Tier suites and cache judgements.
Operational excellence
Owners for each suite, regular refresh from incidents and a dashboard of coverage.
Sustainability
Skip unchanged evaluations and use smaller judges where calibrated.
Network, identity and data
Networking
Evaluation runners sit in a private network with private endpoints to model providers. Adversarial tests that touch tools run against sandboxed copies, never production systems.
Identity and least privilege
Pipeline identities have scoped permissions. Authors cannot override a failed gate. Red-team access to attack corpora is restricted and logged.
Data and encryption
Treat golden sets and attack corpora as classified assets. Avoid real personal information. Version everything and keep results with the model, prompt and guardrail versions tested.
Security and compliance mapping
| Control | How this design addresses it |
|---|---|
| Testing and evaluation | Release gates and evidence support Voluntary AI Safety Standard guardrail 4 and Guidance for AI Adoption test and monitor. |
| Adversarial robustness | Coverage of OWASP LLM risks and MITRE ATLAS techniques. |
| Application control | Only gate-approved signed versions reach production. Essential Eight. |
| Records | Evidence supports third-party assessment (guardrail 9) and ISO/IEC 42001 audits. |
| Change management | Every change tested before release. |
Mappings show how the design supports each control. They are not certifications, and EITSAA does not claim IRAP assessment or compliance for any reference architecture.
Failure modes, targets and scaling
- Target RTOUnder one day. Manual approval with a documented extra review is the fallback.
- Target RPONear zero for evidence, with test assets in version control.
- Scaling and limitsSized by release frequency and suite size.
| Failure mode | Mitigation |
|---|---|
| Golden set no longer reflects real use | Refresh from reviewed production samples and incidents. |
| Judge bias hides regressions | Calibration, spot human review and multiple judges. |
| Thresholds are gamed or relaxed under pressure | Exceptions need named approval and are reported. |
| Attack suites become stale | Update from new research, red-team findings and incidents. |
| Pipeline is slow, so teams bypass it | Fast tier, caching and parallelism. |
Observability and runbooks
- Scores by version and suite over time.
- Attack success rates by category.
- Gate pass rate and exceptions.
- Evaluation cost and duration.
Cost drivers and levers
Evaluation tokens, runner compute, human red-team time and storage.
- Tiered suites.
- Cache unchanged results.
- Use small calibrated judges.
- Sample large sets for routine runs.
Implementation and fit
Phased approach
- Create golden sets and a baseline run for the highest-risk system.
- Add adversarial suites and CI integration.
- Introduce release gates by tier and the evidence store.
- Add human red-team cadence and incident-driven test growth.
Anti-patterns to avoid
- Testing once before launch.
- Only measuring average quality.
- A judge model marking its own work.
- Red-teaming without fixing findings.
- Using real personal data in test sets.
When to use it
- Any LLM, RAG or agent application with real users.
- Suppliers who must show testing evidence in tenders.
When not to
- Throwaway prototypes with no users.
- Projects with no one to own test cases.
More in ai governance and assurance
Observability · incident response
AI observability and incident response: tracing, quality metrics, drift detection, alerts and runbooks
How to see what AI systems are actually doing and respond when they misbehave: end-to-end tracing, quality and safety scoring on sampled traffic, drift and cost detection, alerting tied to runbooks and post-incident learning that feeds testing.
Data governance · privacy · lineage
Responsible AI data governance: classification, consent, lineage and retention for training, retrieval and prompts
How data is governed across the AI lifecycle: sources are classified and assessed for lawful use, consent and purpose are enforced, lineage tracks data into training sets, indexes and prompts, and retention and deletion apply to every derived copy.
AI governance · LLMOps
AI governance and LLMOps platform: model registry, prompt versioning, evaluations, guardrails and responsible AI approvals
The operating system for AI in an agency or enterprise: versioned models and prompts, automated evaluations and red-teaming as release gates, responsible AI approval with a risk register, and production monitoring of quality, drift, safety and cost.
Related insights
Discuss this architecture
Tell us your systems, constraints and timeline. We reply within one business day, and the initial conversation is free.