Ref: https://learn.cantrill.io/courses/1820301/lectures/41301475

DEMO: https://learn.cantrill.io/courses/1820301/lectures/41301476

• Summary
• Contents:

Default S3 Single-Stream Uploads

Diagram: https://github.com/acantril/aws-sa-associate-saac03/blob/main/0700-SIMPLE_STORAGE_SERVICE(S3)/00_LEARNINGAIDS/S3Performance-2.png

• By default, s3:PutObject operation consists of a single stream of a single data blob
• Disadvantages:
  • 👎 If stream fails, upload fails → requires full restart of the upload
    • The bigger the S3 object, the bigger the risk!
  • 👎 Speed & reliability limited by the single stream
  • 👎 Can only upload objects of up to 5GB in size (even if S3 can store objects up to 5TB in size)
• Single-stream uploads are especially bad in global apps
  • Diagram: https://github.com/acantril/aws-sa-associate-saac03/blob/main/0700-SIMPLE_STORAGE_SERVICE(S3)/00_LEARNINGAIDS/S3Performance-1.png
  • The farther away the end-users are from the S3 bucket, the slower the upload, and the higher the risk of upload failing due to network conditions
  • 💡 Bit-Torrent & similar protocols were developed to improve data transfer speed in high-speed NWs
• Two ways to improve performance & reliability for global apps:
  1. S3 Multipart Upload
  2. S3 Transfer Acceleration

S3 Multipart Upload

Diagram: https://github.com/acantril/aws-sa-associate-saac03/blob/main/0700-SIMPLE_STORAGE_SERVICE(S3)/00_LEARNINGAIDS/S3Performance-3.png

• 🔧 Breaks up data blob into smaller parts, s3:putObject uploads those parts in parallel
  • ❗ Can be enabled ONLY for data blobs ≥ 100MB
• Data broken up into a maximum of 10k smaller parts
  • Small parts range between 5MB-5GB in size (last part can be ≤ 5MB)
• 👍 Parts can fail and be restarted in isolation from other parts
• 👍 Improves transfer rate significantly
  • Transfer rate = speeds of all parts (parallel uploads)
  • Better use of bandwidth in NWs
• ‼️ Enable Multipart Upload whenever possible (i.e. ≥100MB uploads)! Benefits are huge over using single-stream upload!
  • 💡 Multipart Upload enabled by default in the AWS Management Console in the Browser UI