• Contents:

AWS Lake Formation - Mini deep-dive

Ref: https://www.udemy.com/course/aws-certified-machine-learning-engineer-associate-mla-c01/learn/lecture/46730185

• 🔧 Manage Data Lakes in S3
  • Can set up a data lake securely and fast (in days → that's actually fast!)
  • Built on top of AWS Glue → Whatever Glue/Spark can do, Lake Formation can also do
  • Architecture Diagram
• Features that facilitate creating and managing data lakes:
  • Loading data & monitoring data flows (data crawlers)
  • Setting up partitions
  • Encryption & managing keys
  • Defining transformation jobs & monitoring them
    • e.g. data cleaning, imputing missing data…
  • Access control (ACLs)
  • Auditing
• Sources can be very different and varied (S3, RDBMS, NoSQL…)
  • They can be on-premises or cloud
• Data lakes can be read directly by Athena, Redshift (Spectrum) and EMR
  • ‼️ Manifests in Athena or Redshift queries are NOT supported!!
• Pricing
  • No cost for Lake Formation itself
  • Billed for underlying services (Glue, S3, EMR, Athena, Redshift…)

Process for building a Data Lake

1. Create an IAM user for Data Analyst
   • Data Analyst will build and use the lake
2. Create AWS Glue connection to your data source(s)
3. Create S3 bucket for data lake
4. Register S3 bucket path in Lake Formation, grant permissions
5. Create DB in Lake Formation for data catalog, grant permissions
6. Use a blueprint for a workflow (e.g. making DB snapshots)
7. Run the workflow
8. Grant SELECT permissions to whoever reads lake (Athena, Redshift Spectrum, etc)

Governed Tables in S3