AI governance and assurance · AI governance · LLMOps
The operating system for AI in an agency or enterprise: versioned models and prompts, automated evaluations and red-teaming as release gates, responsible AI approval with a risk register, and production monitoring of quality, drift, safety and cost.
A reference architecture from EITSAA, not a named client system. Service limits, prices and product features change, so confirm current details before you commit to a design.
Teams change prompts, agents, retrieval settings or models through pull requests. Everything that influences behaviour lives in version control, including evaluation sets and guardrail configuration.
Prompts are versioned in a registry with an owner, purpose, the models they are validated for and a change log, so any production answer can be traced to the exact prompt text used.
Models, whether hosted by a provider or self-hosted, are registered with a model card, licence, data residency, provider terms, known limitations, lineage and the evaluation evidence for that version.
A pipeline runs the evaluation suite: task accuracy, groundedness, refusal behaviour, bias and fairness checks across relevant groups, latency and cost. Judge models are calibrated against human-labelled samples.
The same pipeline runs security and safety tests: prompt injection and jailbreak suites, data leakage probes and tool-abuse scenarios. Guardrail policies (content, topics, PII, grounding checks) are tested alongside the model.
Results are compiled into an evidence pack for the use case: purpose, data, risk tier, evaluation results, residual risks and human oversight design. Low-risk changes can pass automatically. Higher tiers go to a responsible AI board.
Decisions, conditions and review dates are written to the AI risk register against the use case, owner, model and prompt versions in force.
Only approved model, prompt and guardrail versions can be promoted to the production AI gateway. Promotion is a signed pipeline step, and direct calls to providers from applications are blocked.
Production telemetry flows to monitoring: quality scores from sampled traffic, drift in input and output distributions, safety events, user feedback, latency and spend by use case.
Monitoring thresholds raise risks and review tasks in the register. A breach can trigger rollback, a tighter guardrail or a re-evaluation, closing the loop between operation and governance.
| Component | What it does | Why this choice |
|---|---|---|
| Git platform and CI (GitHub, GitLab, Azure DevOps) | Version control, reviews and pipeline execution. | Single audit trail for every change. |
| Prompt registry (Git-backed, Langfuse, MLflow, Azure AI Foundry or Bedrock Prompt Management) | Versioned prompts with ownership and linked evaluations. | Prompts are production code and need release discipline. |
| Model registry (MLflow, SageMaker Model Registry, Azure ML registry, Vertex AI Model Registry) | Model versions, cards, lineage and approval status. | Provenance and approval in one place. |
| Evaluation tooling (promptfoo, Inspect, Ragas, Bedrock Evaluations, Azure AI Evaluation, Vertex Gen AI evaluation) | Automated quality, safety and bias evaluation with human calibration. | Repeatable evidence and regression detection. |
| Guardrails (Amazon Bedrock Guardrails, Azure AI Content Safety, NeMo Guardrails, Llama Guard) | Input and output policy enforcement. | Defence in depth around the model. |
| Approval workflow (ServiceNow, Jira or a purpose-built tool) | Risk-tiered review and sign-off with separation of duties. | Governance that scales with volume. |
| AI risk register | Use cases, risks, controls, owners and review dates. | Single view for executives and assurance teams. |
| Monitoring (OpenTelemetry, Langfuse, Azure Monitor, CloudWatch, Datadog) | Quality, drift, safety, cost and incident signals. | Detects problems that unit tests cannot. |
Risk tiering
Why. Light governance for low-risk use and rigour for high-risk use keeps teams engaged.
Trade-off. Tier definitions need executive agreement and periodic recalibration.
Evaluation as a gate
Why. Prevents silent regressions and creates evidence.
Trade-off. Maintaining evaluation sets is continuing work and needs domain experts.
LLM judges
Why. Scales review cheaply.
Trade-off. Judges have biases and drift, so they cannot be the only control.
One platform or best-of-breed tools
Why. Multi-cloud estates are normal, and tools change quickly.
Trade-off. Integration effort replaces vendor lock-in.
Enforcement point
Why. Governance that cannot be bypassed.
Trade-off. Teams lose ad-hoc freedom, so provide a fast sandbox path for experimentation with synthetic data.
Signed pipeline steps, separation of duties between authors and approvers, least-privilege access to registries and evaluation data, and secrets managed outside code.
Registries and the gateway deployed across zones, rollback to the last approved version and a documented process when monitoring detects harm.
Parallel evaluation runs, sampled online scoring and caching of unchanged test results.
Evaluation tokens are a real cost. Use tiered suites (fast smoke tests per commit, full suites per release) and cached judgements.
Quarterly review of tiers and evaluation sets, post-incident reviews and a published service catalogue of approved models.
Avoid re-running unchanged evaluations, and retire unused models and prompts.
Registries, evaluation runners and the gateway are in private networks with private endpoints to model providers and storage. Evaluation workloads that handle real data run in the same data residency boundary as production. Sandboxes use synthetic or de-identified data and have no route to production data.
Developers, approvers and administrators use SSO with MFA. Pipeline identities are workload identities with scoped permissions. Authors cannot approve their own releases. Access to evaluation data containing personal information is restricted and logged.
Evaluation sets are assets: curate them with domain experts, version them and classify them. Keep test data free of real personal information unless authorised, and record the lineage of any data used for fine-tuning. Retain the evidence pack for each release for the period required by the agency's records obligations.
| Control | How this design addresses it |
|---|---|
| Governance and accountability | Named owners, risk tiers and board sign-off align with the Voluntary AI Safety Standard and AI Ethics Principles. |
| Testing and transparency | Evaluation evidence, model cards and disclosure to users align with AI assurance guidance. |
| Restrict administrative privileges | Separation of duties for author, approver and operator. Essential Eight. |
| Application control | Only approved, signed model and prompt versions can run in production. Essential Eight. |
| Event logging | Release, approval and runtime events retained for investigation and audit. |
Mappings show how the design supports each control. They are not certifications, and EITSAA does not claim IRAP assessment or compliance for any reference architecture.
| Failure mode | Mitigation |
|---|---|
| Silent model update by a provider changes behaviour | Pin versions where possible, run scheduled canary evaluations and alert on score drift. |
| Evaluation set no longer represents real use | Refresh from sampled production traffic with privacy review, and track coverage. |
| Approval becomes a rubber stamp | Evidence packs with specific questions, sampling audits of decisions and rotation of reviewers. |
| Shadow AI bypasses the platform | Block direct provider access at the network and gateway, and make the compliant path the fastest one. |
| A guardrail blocks legitimate work | Measure the false-positive rate, give a tuning path with approval, and monitor appeals. |
Evaluation tokens and compute, platform licences, reviewer time and monitoring data volume.
Operating model · control framework
How an organisation governs every kind of AI: leadership sets direction, an AI register and risk tiers decide how much review each system needs, an approval workflow gates release, platform guardrails enforce policy and assurance closes the loop with evidence.
LLM guardrails · policy engine
A layered guardrail pattern around an LLM application: authenticate and screen the request, redact personal data, retrieve only what the user may see, call the model, then validate grounding and policy on the way out, with every step logged for audit.
AI agents · safety controls
How to let an AI agent act without letting it do harm: a permission broker issues narrow, short-lived tool access, risky code runs in a sandbox, high-impact actions wait for human approval, budgets cap spend and a kill switch stops everything.
Tell us your systems, constraints and timeline. We reply within one business day, and the initial conversation is free.