Data Ingestion and Storage

Types of Data (according to structure)

Type of data	Definition	Characteristics	Examples
Structured	Defined schema	- Easily queryable

Rows and columns
Consistent structure | - Relational DBs
Spreadsheets or CSV files with consistent columns | | Unstructured | No schema nor predefined structure | - Requires heavy preprocessing to query
May come in various formats | - Raw text (e.g. emails, word processing docs…)
Raw media (videos, audio, images…) | | Semi-structured | Some level of structure (tags, hierarchies…) | - Elements might be tagged or categorized
More flexible than structured
Not as chaotic as unstructured | - XML & JSON files
Email headers (structured: date, subject… + unstructured: body…)
Log files with varied formats |

Data Warehouse (DWH) (e.g. Amazon Redshift)
- Centralized repository optimized for analysis (read-heavy operations) where data from different sources is stored in a structured format
Data Lake (e.g. Amazon S3 can be used as data lake)
- Storage repository that holds vast amounts of raw data in its native format (without predefined structure)
- Structured, semi-structured, & unstructured data

Property	Data Warehouse	Data Lake
Schema	Schema-on-write	Schema-on-read
Data Types	Structured	Structured, Unstructured & Semi-structured
Agility	Less agile/flexible	More agile
Processing Pipelines	ETL	ELT
Cost	Optimizations for complex queries are expensive	Cost-effective storage, costs rise when processing lots of data
Use when…	- Structured data sources

Fast & complex queries requirements
BI & analytics | - Big variety of data sources with different structure
Scalable, flexible & cost-effective storage solution
Advanced analytics, ML & data discovery |
💡 Often, organizations use a combination of both, ingesting raw data into a data lake and then processing and moving refined data into a data warehouse for analysis
Data Lakehouse (e.g. AWS Lake Formation with S3 & Redshift Spectrum)
- Hybrid data architecture, tries to provide advantages of both
  - Performance, reliability & capabilities of DWHs
  - Flexibility, scale & low-cost storage of data lakes