A data platform consists of interconnected systems that collect, store, process, govern, and deliver data.

To make this complex environment easier to understand, its capabilities can be divided into logical layers. Each layer represents a group of related functions rather than necessarily representing a separate physical system.

The principal layers are:

1. Data ingestion
2. Data storage and integration
3. Data processing
4. Analysis and consumption
5. Data pipelines and orchestration

Security, governance, metadata, data quality, and observability operate across all these layers.

A Simplified Data Platform

A high-level data flow looks like this:

Source systems
      ↓
Data ingestion
      ↓
Storage and integration
      ↓
Data processing
      ↓
Analysis and consumption

Source systems
      ↓
Data ingestion
      ↓
Storage and integration
      ↓
Data processing
      ↓
Analysis and consumption

Pipeline orchestration coordinates the movement between these layers.

Cross-cutting capabilities protect and monitor the complete platform:

Security · Governance · Metadata · Quality · Lineage · Observability

Security · Governance · Metadata · Quality · Lineage · Observability

Real platforms are not always this linear. Data may be transformed before it is stored, processed repeatedly, or delivered to several destinations simultaneously.

1. Data Ingestion Layer

The data ingestion layer connects to source systems and moves data into the platform.

Common sources include:

• Relational databases
• NoSQL databases
• Business applications
• Files
• APIs
• Websites
• Event streams
• IoT devices
• Application logs
• External providers

The ingestion layer must accommodate different structures, volumes, update frequencies, and access methods.

Responsibilities

The ingestion layer commonly performs the following tasks:

• Connect to source systems
• Authenticate securely
• Extract required data
• Transfer records to the data platform
• Support batch and streaming ingestion
• Detect changes in source data
• Validate incoming formats
• Record ingestion metadata
• Handle retries and failures
• Protect data during transfer

Batch Ingestion

Batch ingestion moves data at scheduled intervals.

Examples include:

• Daily transaction exports
• Hourly database extracts
• Monthly financial files
• Weekly external datasets

Batch processing is appropriate when immediate availability is unnecessary.

Streaming Ingestion

Streaming ingestion continuously receives events or processes them with minimal delay.

Examples include:

• Website clicks
• Financial transactions
• Sensor measurements
• Application logs
• Inventory events
• Vehicle locations

Streaming is useful when the organization must respond quickly, but it introduces additional operational complexity.

Change Data Capture

Change data capture identifies inserts, updates, and deletes in a source system and delivers only those changes.

CDC can reduce:

• Source-system load
• Data-transfer volume
• Processing time
• Update latency

It is commonly used to replicate operational data into analytical platforms.

Ingestion Metadata

The ingestion process should record information such as:

• Source system
• Extraction time
• Number of records received
• Batch or event identifier
• Source schema
• File name
• Pipeline version
• Processing status
• Rejected-record count
• Destination

This metadata supports lineage, auditing, troubleshooting, and data-quality monitoring.

Example Technologies

Technologies used within or near the ingestion layer may include:

• Apache Kafka
• Cloud event-streaming services
• Message queues
• Change-data-capture tools
• Database connectors
• Managed file-transfer services
• Custom API clients

Some processing frameworks, such as Apache Beam and managed execution services such as Google Cloud Dataflow, can process both batch and streaming data. They are broader processing technologies rather than ingestion tools alone.

2. Data Storage and Integration Layer

After data is collected, it must be stored and organized so that it can support operational and analytical workloads.

This layer may include:

• Relational databases
• NoSQL databases
• Data warehouses
• Data marts
• Data lakes
• Lakehouses
• Object storage
• Distributed file systems

Storage Responsibilities

The storage layer should:

• Preserve data reliably
• Support current and historical information
• Accommodate expected growth
• Provide appropriate query performance
• Enforce access controls
• Support backup and recovery
• Manage retention
• Balance performance and cost

Raw and Curated Storage

A platform may maintain several versions of data.

Raw data

Retains records close to their original source representation.

Standardized data

Uses consistent formats, identifiers, and data types.

Curated data

Has been modeled and validated for a particular business or analytical purpose.

Maintaining these layers allows the organization to preserve source information while delivering easier-to-use datasets.

Data Integration

Data integration combines information from different sources into consistent datasets.

Suppose customer information exists in:

• A sales application
• A billing system
• A marketing platform
• A customer-support application

Integration may require:

• Matching customer identifiers
• Standardizing addresses
• Resolving conflicting values
• Removing duplicates
• Converting data types
• Applying shared definitions
• Preserving source lineage

Physical Integration

Physical integration copies and combines data in a shared repository.

Examples include:

• Loading data into a warehouse
• Creating lakehouse tables
• Building a data mart

Logical Integration

Logical integration provides a unified interface without necessarily moving every record into one location.

Examples include:

• Federated queries
• Data virtualization
• Semantic layers
• APIs

Physical integration can provide predictable performance, while logical integration may reduce duplication. Each introduces different trade-offs.

Storage Requirements

A storage architecture should be evaluated for:

Reliability

Data should remain available and recoverable.

Scalability

The system should accommodate increasing data and workload volume.

Performance

Storage structures should support required query and processing patterns.

Security

Sensitive data must be protected through authentication, authorization, encryption, and auditing.

Cost efficiency

Frequently accessed and infrequently used data may require different storage classes or retention strategies.

3. Data Processing Layer

The processing layer validates, cleans, transforms, enriches, and models data.

Processing may occur:

• Before data enters a repository
• After data has been loaded
• Continuously as events arrive
• On a recurring schedule
• On demand

Processing Responsibilities

This layer commonly performs:

• Data validation
• Data cleaning
• Type conversion
• Standardization
• Filtering
• Deduplication
• Aggregation
• Joining
• Enrichment
• Business-rule application
• Data modeling
• Feature creation

Batch Processing

Batch processing handles collections of records together.

It is commonly used for:

• Daily warehouse updates
• Monthly financial calculations
• Historical transformations
• Large analytical workloads

Stream Processing

Stream processing operates on events as they arrive.

It may support:

• Fraud alerts
• Application monitoring
• Real-time recommendations
• Sensor analysis
• Rapid inventory updates

Distributed Processing

When data is too large for one machine, processing may be distributed across multiple nodes.

Technologies such as Apache Spark and Apache Flink divide the workload and coordinate parallel computation.

Distributed processing provides scale but introduces complexity involving:

• Partitioning
• Network communication
• Fault tolerance
• Resource management
• Event ordering
• Late-arriving data

Common Data Transformations

Structuring

Structuring changes the form or schema of data.

Examples include:

• Reordering fields
• Flattening nested JSON
• Splitting one field into several columns
• Combining multiple fields
• Joining datasets
• Creating aggregates

Standardization

Standardization converts values into consistent representations.

Examples include:

• Converting dates to ISO format
• Standardizing country codes
• Converting currencies
• Normalizing units
• Applying consistent category labels

Normalization

In relational database design, normalization organizes data into related tables to reduce redundancy and update inconsistencies.

For example, customer details and transaction records may be stored in separate tables connected by a customer key.

Normalization should not be described simply as removing unused data. It is a formal approach to organizing dependencies among attributes.

Denormalization

Denormalization deliberately combines or duplicates selected data to simplify queries or improve read performance.

It is commonly used in:

• Data warehouses
• Reporting tables
• Analytical aggregates
• Document databases

Denormalization trades increased storage and update complexity for easier or faster retrieval.

Data Cleaning

Data cleaning addresses irregularities such as:

• Missing values
• Duplicates
• Invalid types
• Inconsistent formats
• Impossible values
• Broken relationships
• Unexpected categories

Cleaning rules should reflect the business meaning of the data rather than making arbitrary changes solely to eliminate errors.

Processing Technologies

Processing can be performed using:

• SQL
• Python
• R
• Java
• Data transformation frameworks
• Distributed-processing engines
• Cloud data-processing services
• ETL and ELT platforms
• Data preparation tools

The appropriate technology depends on:

• Data size
• Data structure
• Batch or streaming requirements
• Latency
• Team expertise
• Existing platforms
• Cost
• Governance requirements

Spreadsheets may be useful for small, manual investigations, but they are generally unsuitable for large, recurring production transformations.

Storage and Processing May Overlap

The architecture’s layers are logical rather than always physically separate.

Database processing

A relational database can store data and execute SQL transformations within the same system.

Warehouse processing

A cloud warehouse may load raw data and perform transformations using its own computing engine.

Lake processing

A data lake may store files in object storage while separate engines process those files.

Stream processing

Events may be transformed before they are written to long-term storage.

The implementation depends on the selected architecture.

ETL and ELT Placement

ETL

With ETL, major transformations occur before data reaches the target repository:

Extract → Transform → Load

Extract → Transform → Load

The processing layer therefore appears before or between storage systems.

ELT

With ELT, data is first loaded and then transformed within the target platform:

Extract → Load → Transform

Extract → Load → Transform

The storage and processing layers become more closely connected.

A platform may use both patterns for different datasets.

4. Analysis and Consumption Layer

The analysis and consumption layer delivers data to people, applications, and services.

Consumers may include:

• Data analysts
• Business intelligence analysts
• Data scientists
• Business stakeholders
• Machine-learning systems
• Operational applications
• External partners
• Customers

Different consumers require different access methods and levels of detail.

Business Intelligence

Business users may access data through:

• Dashboards
• Scheduled reports
• Interactive visualizations
• Key performance indicators
• Alerts
• Self-service analytical tools

Popular categories of tools include:

• Business intelligence platforms
• Reporting systems
• Spreadsheet applications
• Semantic modeling tools

Analytical Access

Analysts may use:

• SQL clients
• Notebooks
• Python
• R
• Statistical applications
• Visualization libraries

Data Science Access

Data scientists may require:

• Detailed historical data
• Training and validation datasets
• Feature stores
• Notebooks
• Distributed computing
• Model-development environments

Application Access

Applications may consume data through:

• APIs
• Database connections
• Event streams
• Data services
• Export files
• Caches

Applications often require lower latency and stricter service guarantees than human analytical users.

User Interfaces

Interfaces translate data into forms appropriate for different audiences.

Examples include:

• Executive dashboards
• Operational reports
• Analytical notebooks
• Data portals
• Search interfaces
• Embedded application analytics

Jupyter Notebooks, Python, and R are analytical development environments or languages rather than conventional drag-and-drop BI dashboards, although they can generate reports and visualizations.

Semantic Layer

A semantic layer presents technical data using consistent business concepts.

It may define:

• Revenue
• Active customer
• Conversion rate
• Product category
• Reporting period
• Geographic hierarchy

This prevents every dashboard or analyst from independently implementing the same metric.

The semantic layer can sit between storage and consumption, making governed data easier to use.

5. Data Pipeline and Orchestration Layer

A data pipeline represents the complete movement of data from source to destination.

Because pipelines coordinate ingestion, storage, transformation, and delivery, they overlay several architectural layers.

Source
  ↓
Ingestion task
  ↓
Raw storage
  ↓
Transformation task
  ↓
Curated storage
  ↓
Quality check
  ↓
Dashboard or application

Source
  ↓
Ingestion task
  ↓
Raw storage
  ↓
Transformation task
  ↓
Curated storage
  ↓
Quality check
  ↓
Dashboard or application

Pipeline Responsibilities

The pipeline layer may provide:

• Scheduling
• Task dependencies
• Workflow execution
• Retry handling
• Failure notification
• Parameter management
• Logging
• Metadata capture
• Data-quality checks
• Backfill processing
• Recovery
• Deployment management

Orchestration

Orchestration coordinates when and in what order tasks run.

For example:

1. Wait for a source file.
2. Validate the file.
3. Load it into raw storage.
4. Run transformations.
5. Test the curated table.
6. Publish the result.
7. Refresh the dashboard.
8. Notify the team if a task fails.

Apache Airflow is a widely used workflow orchestrator. It coordinates tasks but is not itself the primary large-scale data-processing engine.

Processing services such as Google Cloud Dataflow have a different purpose: they execute batch and streaming data-processing pipelines.

Cross-Cutting Capabilities

Several capabilities apply across the entire platform rather than belonging to only one layer.

Security

Security controls include:

• Authentication
• Role-based access
• Encryption
• Network restrictions
• Secrets management
• Audit logging
• Data masking
• Incident response

Security should be designed throughout the platform rather than added only at the user-interface layer.

Data Governance

Governance defines:

• Ownership
• Stewardship
• Business definitions
• Acceptable use
• Retention
• Accountability
• Quality expectations
• Access policies

Metadata Management

Metadata describes:

• Sources
• Schemas
• Owners
• Refresh times
• Processing history
• Quality
• Classifications
• Lineage

Metadata allows users and systems to understand the platform’s assets.

Data Lineage

Lineage shows how data moves and changes:

Source transaction
        ↓
Raw orders table
        ↓
Transformation
        ↓
Sales fact table
        ↓
Revenue dashboard

Source transaction
        ↓
Raw orders table
        ↓
Transformation
        ↓
Sales fact table
        ↓
Revenue dashboard

Lineage supports troubleshooting, auditing, impact analysis, and trust.

Data Quality

Quality controls may evaluate:

• Completeness
• Accuracy
• Validity
• Consistency
• Timeliness
• Uniqueness

Quality rules should operate during ingestion, processing, storage, and publication.

Observability

Observability helps teams understand whether the platform and its data are functioning correctly.

Signals may include:

• Pipeline failures
• Processing duration
• Data freshness
• Record volume
• Schema changes
• Resource consumption
• Query performance
• Data-quality anomalies

A pipeline can finish successfully while still producing incorrect data, so both technical and data-level monitoring are necessary.

Reference Architecture

A more complete logical architecture looks like this:

DATA SOURCES
Databases · APIs · Files · Applications · Events · IoT
                         ↓
INGESTION
Batch · Streaming · CDC · File Transfer · API Collection
                         ↓
STORAGE AND INTEGRATION
Databases · Warehouse · Lake · Lakehouse · Data Marts
                         ↓
PROCESSING
Validation · Cleaning · Transformation · Enrichment · Modeling
                         ↓
ANALYSIS AND CONSUMPTION
SQL · BI · Dashboards · Notebooks · ML · APIs · Applications

PIPELINES AND ORCHESTRATION
Coordinate movement and processing across the layers

CROSS-CUTTING CONTROLS
Security · Governance · Metadata · Lineage · Quality · Observability

DATA SOURCES
Databases · APIs · Files · Applications · Events · IoT
                         ↓
INGESTION
Batch · Streaming · CDC · File Transfer · API Collection
                         ↓
STORAGE AND INTEGRATION
Databases · Warehouse · Lake · Lakehouse · Data Marts
                         ↓
PROCESSING
Validation · Cleaning · Transformation · Enrichment · Modeling
                         ↓
ANALYSIS AND CONSUMPTION
SQL · BI · Dashboards · Notebooks · ML · APIs · Applications

PIPELINES AND ORCHESTRATION
Coordinate movement and processing across the layers

CROSS-CUTTING CONTROLS
Security · Governance · Metadata · Lineage · Quality · Observability

Example: Retail Data Platform

Consider a retailer with physical stores and an online business.

Ingestion Layer

The platform collects:

• Point-of-sale transactions
• Online orders
• Website clickstreams
• Inventory updates
• Customer-support records
• Supplier files

Storage and Integration Layer

• Raw events are stored in a data lake.
• Operational data is copied from databases.
• Customer and product identifiers are standardized.
• Curated tables are stored in a warehouse.

Processing Layer

Pipelines:

• Remove duplicates
• Standardize time zones
• Calculate revenue
• Match customers across systems
• Aggregate daily sales
• Create machine-learning features

Analysis and Consumption Layer

• Managers view dashboards.
• Analysts query warehouse tables.
• Data scientists build demand forecasts.
• Applications retrieve recommendations through APIs.

Cross-Cutting Controls

• Sensitive customer data is classified.
• Access is limited by role.
• Quality tests validate revenue totals.
• Lineage connects dashboards with source transactions.
• Monitoring detects late or failed pipelines.

Principles for Designing the Layers

Keep interfaces clear

Each layer should provide documented inputs and outputs.

Select tools according to function

Do not use a workflow orchestrator as a processing engine or a BI tool as a production integration platform.

Design for failure

Sources, networks, jobs, and services will eventually fail. Pipelines need retry, recovery, and alerting behavior.

Capture metadata automatically

Record sources, schemas, processing times, counts, and lineage wherever possible.

Apply security throughout

Sensitive data must remain protected during ingestion, storage, processing, and access.

Avoid unnecessary complexity

A small platform may not require separate technology for every conceptual layer.

Design around consumer needs

Architecture should deliver data with the freshness, quality, structure, and performance its consumers require.

Key Takeaways

• A data platform can be divided into ingestion, storage and integration, processing, and consumption layers.
• These are logical functions and may not correspond to separate physical systems.
• The ingestion layer moves batch and streaming data from sources into the platform.
• Storage systems preserve raw, standardized, and curated data.
• Integration combines data from separate sources into consistent datasets.
• The processing layer validates, cleans, transforms, enriches, and models data.
• The consumption layer delivers information through SQL, reports, dashboards, notebooks, APIs, and applications.
• Pipelines and orchestration coordinate work across multiple layers.
• ETL transforms data before loading, while ELT transforms data after loading.
• Security, governance, metadata, quality, lineage, and observability apply across the complete architecture.
• Architecture should be based on functional requirements rather than the popularity of particular tools.

Conclusion

A layered model provides a useful way to understand a modern data platform.

Data enters through the ingestion layer, is preserved and integrated through storage systems, becomes usable through processing, and reaches consumers through analytical and application interfaces. Pipelines coordinate this movement, while governance and operational controls protect the entire system.

The layers may overlap in implementation, but separating their responsibilities helps teams design platforms that are easier to scale, secure, monitor, and maintain.

One-sentence summary: A modern data platform connects ingestion, storage, integration, processing, and consumption through coordinated pipelines supported by security, governance, metadata, quality, and observability.