Instrumentation
OpenTelemetry first.
Why. Portable, consistent across languages.
Trade-off. Some auto-instrumentation is less polished than vendor agents.
Cloud platforms · AWS and Azure
Observability stack: logs, metrics, traces, SLOs and alerting
One way to see every workload: OpenTelemetry instrumentation, central storage per signal, SLO-based alerting that wakes people only for user impact, and runbooks linked from every alert.
A reference architecture from EITSAA, not a named client system. Service limits, prices and product features change, so confirm current details before you commit to a design.
- ACSC Essential Eight
- SA ISMF logging expectations
- Australian regions
Architecture
- Compute and containers
- SaaS and business systems
- Databases and data
- Monitoring, audit and analytics
- Network and edge
- Request or data flow
- Asynchronous or optional
- Boundary: region, account, network or trust zone
Request and data flow
Applications emit traces, metrics and structured logs through OpenTelemetry SDKs with a consistent service name, environment and version.
The collector samples traces, scrubs personal information and secrets, and routes each signal to the right store.
Platform services also send native logs and metrics, such as ALB access logs, Azure diagnostic settings and database slow-query logs.
SLIs (availability, latency, error rate) feed SLO definitions, with alerts on burn rate across short and long windows.
Only user-impacting alerts page the on-call engineer. Everything else becomes a ticket or a dashboard item.
Each alert links to its runbook, dashboards and the recent deployment, and incidents are recorded for review.
Components and why they were chosen
| Component | What it does | Why this choice |
|---|---|---|
| OpenTelemetry SDK and Collector | Vendor-neutral instrumentation and a pipeline for filtering and routing. | Avoids lock-in and centralises scrubbing. |
| Amazon CloudWatch / Azure Monitor | Native metrics, logs, alarms and dashboards. | Zero-setup coverage for managed services. |
| Amazon Managed Prometheus and Managed Grafana (or Azure Managed Grafana) | Metrics storage and cross-account dashboards. | Good for Kubernetes and for one view across clouds. |
| AWS X-Ray or Application Insights | Distributed tracing and service maps. | Find latency sources across services. |
| CloudWatch Logs Insights, OpenSearch or Log Analytics (KQL) | Query logs. | Choose by existing skills and volume. |
| SLO tooling (CloudWatch Application Signals, Grafana SLO, Nobl9) | Define error budgets and burn alerts. | Alerts tied to user outcomes cut noise. |
| On-call platform | Schedules, escalation and acknowledgement. | Humans need a reliable path. |
| Runbooks (Git, Confluence or wiki) | Step-by-step diagnostics and recovery. | Reduce time to restore and dependence on specific people. |
Design decisions and trade-offs
Alerting
SLO burn-rate alerts rather than CPU and memory thresholds.
Why. Pages should mean users are hurt.
Trade-off. Needs agreed SLOs, which is a conversation worth having.
Log retention
Hot for 30 days, archive to S3 or Storage for the policy period.
Why. Controls cost while meeting audit needs.
Trade-off. Archived logs are slower to query.
Placement
Central observability account with cross-account sharing.
Why. One pane of glass and protected data.
Trade-off. Cross-account permissions to maintain.
Well-architected pillars
Security
Scrub personal information before storage, restrict log access, and treat logs as sensitive data.
Reliability
Telemetry paths are buffered, with an independent heartbeat that alerts if telemetry stops.
Performance
Head and tail sampling keeps traces affordable and fast to query.
Cost optimisation
Ingestion volume is the main driver. Control cardinality and verbosity.
Operational excellence
Alert review monthly: delete or fix every alert that was not actionable.
Sustainability
Collect what you will use, retain what you must.
Network, identity and data
Networking
Collectors send over private endpoints or the cloud's backbone. Dashboards are reachable only via SSO. Cross-account sharing uses IAM roles, not network exposure.
Identity and least privilege
Read access to production telemetry is a role with SSO and MFA, and access to logs with personal data is further restricted. Alert tooling uses scoped API keys in a secret manager.
Data and encryption
Never log credentials, tokens or full personal records. Use field-level redaction in the collector, and classify log groups with matching retention.
Security and compliance mapping
| Control | How this design addresses it |
|---|---|
| Event logging | Centralised, protected and time-synchronised logs support ISM event logging guidance and incident response. |
| Restrict administrative privileges | Log deletion and retention changes restricted and alerted. Essential Eight. |
| Incident response | Alerts, runbooks and post-incident review support ISMF incident management expectations. |
Mappings show how the design supports each control. They are not certifications, and EITSAA does not claim IRAP assessment or compliance for any reference architecture.
Failure modes, targets and scaling
- Target RTOObservability is restored before other systems in a disaster, in under an hour
- Target RPOMetrics minutes; logs per retention policy
- Scaling and limitsMetric cardinality and log volume are the limits. Budget and review them.
| Failure mode | Mitigation |
|---|---|
| Telemetry pipeline stops silently | Dead-man's switch alert if no data arrives. |
| Alert fatigue | SLO-based paging and monthly alert review. |
| Cost explosion from log volume | Ingestion alarms, sampling and log level policy. |
| Secrets or personal data in logs | Collector scrubbing rules and scheduled log scanning. |
Observability and runbooks
- Meta-monitoring: collector health, ingestion lag and dropped spans.
- Alert quality metrics: pages per week, percent actionable, time to acknowledge.
Cost drivers and levers
Log ingestion and retention, custom metric count, trace volume, dashboards and queries.
- Reduce debug logging in production.
- Tail-sample traces keeping errors and slow requests.
- Drop high-cardinality labels.
- Archive logs to object storage.
Implementation and fit
Phased approach
- Standard logging format and correlation IDs.
- Collect golden signals for each service and define SLOs.
- Replace threshold alerts with burn-rate alerts and add runbooks.
- Add tracing and cross-account dashboards.
Anti-patterns to avoid
- Alerting on every metric.
- Logging whole request bodies.
- Dashboards nobody owns.
- No runbook linked to the alert.
When to use it
- Any production workload, scaled to its criticality.
- Managed service contracts that need evidence of service levels.
When not to
- Building a custom monitoring platform when managed services meet the need.
More in cloud platforms
AWS
Static website on S3, CloudFront, WAF and Route 53
The default for marketing sites, documentation and public information: a private S3 origin behind CloudFront with WAF, managed TLS and a pipeline that deploys with short-lived credentials.
AWS
Serverless API with API Gateway, Lambda, DynamoDB and Cognito
A scale-to-zero API for web and mobile apps. Cognito authenticates, API Gateway enforces, Lambda runs the business logic, DynamoDB stores state and SQS or EventBridge decouple the slow work.
AWS
Three-tier web application on ALB, ECS Fargate and Aurora
The workhorse for line-of-business web apps: public load balancer, containers across three Availability Zones, a private Aurora PostgreSQL cluster, ElastiCache for sessions and S3 for files.
Related insights
Discuss this architecture
Tell us your systems, constraints and timeline. We reply within one business day, and the initial conversation is free.