Ref: https://learn.cantrill.io/courses/1820301/lectures/41301408

• Summary:
• Contents:

Amazon SQS (Simple Queue Service) - Overview

• 🔧 Fully-managed message queues
  • Producers send messages to queue, consumers poll queue to get messages
  • Queues can be Standard or FIFO when it comes to message ordering guarantees:
    1. Standard → best efforts when it comes to order (can potentially arrive out of order)
    2. FIFO = First In First Out → order is guaranteed
  • Small messages (up to 256kB in size, like in SNS)
    • Easier to manage & scale small messages
    • If large data needed → messages should link to data
• Managed by AWS
  • Serverless
  • Public service → public endpoints
  • Regionally resilient → HA, scalable
  • Highly performant
  • 💡 SQS was the first AWS service released to the public, back in 2004!
• Polling = checking if there are messages in a queue
  • ‼️ Received messages from polling are NOT automatically deleted from the queue, instead they are hidden!!
    • Messages deleted permanently from queues when…
      1. Consumer who polled the queue explicitly deletes message from queue (because processing was successful)
      2. VisibilityTimeout expires, at which point message returns to being visible (consumer couldn't process message successfully on time)
         • 👍 Reliable → same or different consumer can retry processing that message
• Problematic messages (e.g. failed to process 5+ times) can be sent to Dead-Letter Queues (DLQs)
  • Different processes can be used for problematic messages
• Use cases → Queues are great for:
  • Decoupling components
  • Scaling → ASGs can scale and Lambdas be invoked based on queue length
    • Allows building complex worker pool style architectures

Amazon SQS - Example Worker Pool Architecture

Diagram: https://github.com/acantril/aws-sa-associate-saac03/blob/main/1600-SERVERLESS_and_APPLICATION_SERVICES/00_LEARNINGAIDS/SQS-1.png

• 💡 We return CatTube example app from previous Architecture Deep Dive page, specifically to section asynchronous queues

1. ASG Web Pool
   • Customers upload master videos to WEB
   • ASG stores master video in S3 bucket Master, then places a message with the link to the video in an SQS queue
   • Customers may query WEB to retrieve transcoded videos from S3 bucket Transcode
   • ASG scales based on CPU load (i.e. how much it's queried by customers)
2. ASG Worker Pool
   • ASG polls SQS queue looking for messages and scales based on queue length
     • If many messages → launch instances
     • If few messages → terminate instances (potentially scale down to 0)
   • Message retrieved from queue, master video retrieved from Master , video processed to different sizes and stored in Transcode bucket
   • If message processing or instances fail, queue message reappears on queue after VisibilityTimeout expires, and it can be retried again

💡 SQS queue effectively decouples Web Pool from Worker Pool (neither care about health nor responses from the other component)

Fanout Architecture with SNS and SQS

Diagram: https://github.com/acantril/aws-sa-associate-saac03/blob/main/1600-SERVERLESS_and_APPLICATION_SERVICES/00_LEARNINGAIDS/SQS-2.png