Table format
Parquet with Apache Iceberg for tables that need updates and time travel.
Why. Open formats keep engines interchangeable and support upserts.
Trade-off. Table maintenance (compaction, snapshot expiry) must be scheduled.
Cloud platforms · AWS and Microsoft
Data lake and BI with S3, Glue, Athena, Redshift and Power BI or QuickSight
A governed analytics platform: raw, curated and consumption layers on object storage, a catalogue with column-level access control and a semantic layer that business users can trust.
A reference architecture from EITSAA, not a named client system. Service limits, prices and product features change, so confirm current details before you commit to a design.
- Australian Privacy Principles (APP 11)
- ACSC Essential Eight
- SA ISMF and ACSC ISM alignment
- Sydney (ap-southeast-2)
Architecture
- SaaS and business systems
- Databases and data
- Object and file storage
- Monitoring, audit and analytics
- Security and identity
- Request or data flow
- Asynchronous or optional
- Boundary: region, account, network or trust zone
Request and data flow
Sources are ingested on a schedule or by change data capture: AppFlow for SaaS, DMS for databases and Firehose or SFTP transfer for files.
Data lands unchanged in the raw zone with the load timestamp and source recorded. Raw is append-only with versioning and Object Lock for audit-critical feeds.
Glue jobs or dbt validate, de-duplicate, conform keys and mask or tokenise personal data using documented rules.
Curated tables are written as partitioned Parquet or Iceberg tables and registered in the Glue Data Catalog.
Lake Formation grants table, column and row-level permissions to roles, so analysts see only what policy allows.
Athena queries the lake serverlessly. Redshift serves heavy concurrent dashboards and pre-modelled marts.
Power BI or QuickSight connects through a gateway or VPC connection to a governed semantic layer with row-level security.
Components and why they were chosen
| Component | What it does | Why this choice |
|---|---|---|
| Amazon S3 (raw, curated, consumption) | Durable, cheap storage with versioning and lifecycle. | Separates storage from compute and avoids lock-in to one engine. |
| AWS Glue and Data Catalog | Serverless ETL and the shared schema catalogue. | Catalogue is shared by Athena, Redshift Spectrum and EMR. |
| AWS Lake Formation | Central fine-grained access control and tagging. | One policy point instead of bucket policies per team. |
| Amazon Athena | Serverless SQL over S3, priced per data scanned. | Ideal for ad hoc work. Partitioning and Parquet cut cost sharply. |
| Amazon Redshift (Serverless or provisioned) | MPP warehouse for high-concurrency BI. | Use when dashboards need consistent sub-second performance. |
| AWS DMS and Amazon AppFlow | Database CDC and SaaS ingestion. | Managed connectors reduce custom code. |
| Power BI or Amazon QuickSight | Dashboards, semantic model and row-level security. | Choose Power BI where the organisation already runs Microsoft 365 and Fabric. |
| Microsoft Fabric (alternative) | OneLake, Lakehouse, Warehouse and Power BI in one SaaS. | Reasonable alternative for Microsoft-centric organisations, with Australian capacity regions. |
Design decisions and trade-offs
Query engine
Athena for exploration, Redshift for governed marts.
Why. Pay per query for the long tail, provisioned speed for dashboards.
Trade-off. Two engines to govern; Lake Formation keeps rules consistent.
Personal information
Tokenise or mask at the curated boundary.
Why. Most analysis does not need identity.
Trade-off. Re-identification workflows need a controlled, audited path.
Transformation
SQL-first modelling (dbt on Athena or Redshift) with tests.
Why. Reviewable logic and data quality gates.
Trade-off. Needs analytics engineering discipline.
Well-architected pillars
Security
KMS-encrypted zones, Lake Formation column and row security, private connectivity and CloudTrail data events on sensitive buckets.
Reliability
S3 durability, idempotent pipelines, replayable raw zone and orchestration retries with alerts.
Performance
Partitioning, columnar formats, compaction and materialised views for dashboards.
Cost optimisation
Athena scan volume, Redshift capacity and Glue DPU hours. Governance prevents duplicate pipelines.
Operational excellence
Orchestration (Step Functions or Airflow), data quality tests and lineage, with owners per dataset.
Sustainability
Compression, lifecycle to colder tiers and pruning of unused datasets.
Network, identity and data
Networking
Glue and Redshift run inside the VPC with S3 gateway and Glue interface endpoints. BI tools connect through a data gateway or QuickSight VPC connection, not the public internet.
Identity and least privilege
Access is by role and LF-tag (for example classification=confidential). Analysts get IAM Identity Center permission sets that map to LF roles. Service roles per pipeline.
Data and encryption
Classify at ingest: public, internal, sensitive, personal. Use SSE-KMS with separate keys for raw and curated, tag columns containing personal information and apply masking. Retention and deletion rules follow the privacy policy and APP 11 destruction obligations.
Security and compliance mapping
| Control | How this design addresses it |
|---|---|
| Personal information handling | Masking, column permissions and retention schedules support APP 6, 11 and 12 (access and correction). |
| Regular backups | Versioning, replication and Object Lock for critical feeds. Essential Eight. |
| Restrict administrative privileges | Data admins separated from data consumers. Essential Eight. |
| Audit | Lake Formation and CloudTrail data events give evidence of who queried what. |
Mappings show how the design supports each control. They are not certifications, and EITSAA does not claim IRAP assessment or compliance for any reference architecture.
Failure modes, targets and scaling
- Target RTOHours to rebuild curated layers from raw
- Target RPOPer feed: near real time for CDC, daily for batch files
- Scaling and limitsAthena and S3 scale to petabytes. Redshift concurrency scaling or Serverless handles dashboard peaks.
| Failure mode | Mitigation |
|---|---|
| Bad source data corrupts reports | Raw is immutable. Rebuild curated from raw after the fix, with quality tests that block publish. |
| Runaway Athena query | Workgroup scan limits and cost alarms. |
| Schema drift from a SaaS source | Schema evolution rules and alerts on new or removed columns. |
| Permission mistake exposes a column | Tag-based policies, automated tests for sensitive tags and access reviews. |
Observability and runbooks
- Pipeline success and freshness SLAs per dataset.
- Row-count and null-rate quality tests with alerts.
- Cost per workgroup and per team, and a monthly review of unused tables.
Cost drivers and levers
Storage volume, Athena data scanned, Redshift hours or RPUs, Glue DPU hours, BI licences.
- Partition on the columns people actually filter by.
- Convert to Parquet and compact small files.
- Pause or autoscale Redshift.
- Lifecycle raw data to Infrequent Access or Glacier tiers.
Implementation and fit
Phased approach
- Pick two or three high-value questions and the sources they need.
- Stand up the lake, catalogue and governance with one pipeline end to end.
- Model the first mart and dashboard with the business owner.
- Add sources by demand and retire spreadsheet reports as trust grows.
Anti-patterns to avoid
- Dumping everything in S3 with no catalogue or owner.
- Giving analysts direct bucket access.
- Building a warehouse before you know the questions.
- Duplicate extracts per team.
When to use it
- Several source systems and recurring reporting needs.
- Need for governed access to sensitive data.
- AI and ML use cases that will need clean data.
When not to
- One source and a few reports (a database and Power BI may be enough).
- Organisations without a data owner.
More in cloud platforms
AWS and Azure
Disaster recovery: backup and restore, pilot light and warm standby
Choose recovery capability by business impact, not by habit. Three patterns with honest RTO and RPO, a cross-Region data path, an immutable backup vault and a failover that is rehearsed rather than hoped for.
AWS and Azure
Observability stack: logs, metrics, traces, SLOs and alerting
One way to see every workload: OpenTelemetry instrumentation, central storage per signal, SLO-based alerting that wakes people only for user impact, and runbooks linked from every alert.
AWS
Static website on S3, CloudFront, WAF and Route 53
The default for marketing sites, documentation and public information: a private S3 origin behind CloudFront with WAF, managed TLS and a pipeline that deploys with short-lived credentials.
Related insights
Discuss this architecture
Tell us your systems, constraints and timeline. We reply within one business day, and the initial conversation is free.