Amazon MSK 101
Ref: https://www.udemy.com/course/aws-certified-machine-learning-engineer-associate-mla-c01/learn/lecture/45356759
- 🔧 Kafka-aaS in AWS
- Apache Kafka = open-source distributed event streaming platform
- Popular for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications
- 💡 No need to know Kafka for exam, but understanding differences between MSK and Kinesis Data Streams is needed
- Kafka at a high level
Architecture (Provisioned Mode)
- CRUD Kafka clusters
- Private service
- MSK creates & manages Kafka broker nodes & Zookeeper nodes
- MSK cluster deployed in your VPC
- Data stored on EBS volumes
- MSK Architecture Diagram
Security
- Encryption:
- Optional in-flight using TLS with brokers and/or clients
- At rest: EBS encryption with KMS
- NW: Authorize specific SGs for your Kafka clients
- Authentication (AuthN) & Authorization (AuthZ): (who can R/W to which topics?)
- Built-in Kafka methods:
- Mutual TLS (AuthN) + Kafka ACLs (AuthZ)
- SASL/SCRAM (AuthN) + Kafka ACLs (AuthZ)
- OR AWS method:
- IAM Access Control (AuthN + AuthZ)
- Diagram
Monitoring
- CW Metrics
- Prometheus
- Broker Log Delivery
MSK Connect