Ref: https://learn.cantrill.io/courses/1820301/lectures/41301509 and https://www.udemy.com/course/aws-certified-solutions-architect-associate-saa-c03/learn/lecture/18078323

• Summary:
• Contents:

Lustre File System

• 🔧 Designed for High Performance Computing (HPC)
  • Parallel data processing at big scale
  • Throughput: 100s of GB/s; Latency: <1ms
  • Use cases: ML, Big Data, Financial Modeling…
• For Linux clients (POSIX permissions)
  • 💡 Lustre = “Linux cluster”
• Data is processed inside File System → Data “lives” in FS
• Lustre splits data up when storing it to disks (storage volumes)
  • Metadata Storage Targets (MSTs) → metadata (file names, timestamps, permissions…)
  • Object Storage Targets (OSTs) → data objects
    • OST size = 1.17TiB
    • Splitting data across OSTs achieves high performance
• Data can be backed in a repository
  • In the case of FSx for Lustre, the repository is an S3 bucket
• ‼️ The Lustre FS is completely separate from the repository/S3 bucket!
  • ❗ Data loaded lazily from the repository
    • All files from repository are always visible to FS, but not necessarily loaded
    • Files loaded from repository into FS when needed (e.g. when a file is accessed, if it's not already in FS, it is loaded from repo)
  • ❗ Data changes in FS are NOT immediately synced to repository
    • Must export back to repository with hsm_archive command
  • Diagram: Lustre FS ≠ Lustre Repository (S3 Bucket)

FSx for Lustre - Key Concepts

• 🔧 Fully-Managed Lustre FS in AWS
  • ‼️ ONLY Single-AZ deployment!!
    • One AZ so it can achieve high performance
• Optional Lustre repository: S3 bucket → Seamless integration with S3
  • ❗ Can “read S3” as a FS (though FSx)
  • Can write the output of computations back to S3 (through FSx)
  • Can configure Manual or Automatic (0-35 day retention) backups to S3
  • 💡 Useful since data is only in one AZ
• Two deployment types:
  1. Scratch
     • 🔧 Highest performance, temporary storage
     • ❗ No data replication, no HA
       • 💡 Larger FS → more servers, more disks → more risk of failure!
     • Optimized for short-term/temporary workloads
       • Less costs too, since there's no data replication or HA
     • Diagram
  2. Persistent
     • 🔧 Persistent storage
     • Data replication (in one AZ!) → HA (in one AZ!)
       • Self-healing capabilities if HW failure
         • Replaces failed files within minutes
       • 💡 If whole AZ fails, you still lose all the data
     • Optimized for long-term workloads
     • Diagram
  • Extra reading on Scratch vs Persistent: https://aws.amazon.com/blogs/storage/persistent-storage-for-high-performance-workloads-using-amazon-fsx-for-lustre/
• Performance:
  • Baseline performance based on FS size
    • FS size: minimum 1.2TiB, then increments of 2.4TiB
    • Scratch: Base 200MB/s per TiB of storage
    • Persistent offers 3 base levels: 50MB/s, 100MB/s, or 200MB/s per TiB of storage
  • Burst up to 1300MB/s per TiB (credit system akin to EBS volumes)
  • Extra ref: https://docs.aws.amazon.com/fsx/latest/LustreGuide/performance.html#fsx-aggregate-perf

FSx for Lustre - Example Architecture