Simple Queue Serivice (SQS) Vs Simple Notification Service Vs Kinesis data Streams/Firehose - Overview.
Scope:
- Intro,
- Overview,
- Use Case Scenarios,
- Detailed Comparison,
- Best Fit Summary,
- when to Combine Them,
- Insights.
Intro:
- Here's twtech comparison of Amazon SQS, SNS, and Kinesis, highlighting their key differences and best use cases:
Overview
|
Feature |
SQS
(Simple Queue Service) |
SNS
(Simple Notification Service) |
Kinesis |
|
Type |
Message Queue (Pull) |
Pub/Sub Notification (Push) |
Data Streaming Platform |
|
Pattern |
Point-to-Point |
Publish-Subscribe |
Real-time data streaming |
|
Message Flow |
Producer → Queue → Consumer. |
Publisher → Topic → Subscribers. |
Producer → Stream → Consumer. |
Use Case Scenarios
|
Use
Case |
Recommended
Service |
|
Decoupling microservices or
systems |
SQS |
|
Broadcasting messages to multiple
systems (e.g., SMS, Email, Lambda) |
SNS |
|
Real-time analytics or processing
high-throughput data streams (e.g., logs, IoT) |
Kinesis |
Detailed Comparison
|
Feature |
SQS |
SNS |
Kinesis |
|
Delivery Type |
Pull-based |
Push-based |
Pull or Enhanced Fan-out |
|
Delivery Guarantee |
At-least-once (FIFO supports
exactly-once) |
At-least-once |
At-least-once (manual deduping
needed) |
|
Ordering |
FIFO queues preserve order |
No ordering guarantee |
Order per shard |
|
Durability |
4-day retention by default (14-day
max) |
Short-lived (no storage) |
24 hrs to 7 days retention |
|
Latency |
Low |
Very low |
Low (milliseconds) |
|
Throughput |
High (scales with batch, FIFO has
limits) |
Very high |
Extremely high (designed for
large-scale ingestion) |
|
Fan-out |
No (1-to-1 delivery) |
Yes (1-to-many) |
Yes (with enhanced fan-out) |
|
Retry Mechanism |
Yes (DLQs supported) |
Yes (retry policies) |
Yes (checkpointing via consumer
apps) |
|
Cost Model |
Per request & payload size |
Per request & delivery attempt |
Per shard-hour + PUT payload units |
Best Fit Summary
|
Use
Case |
Best
Option |
|
Decoupled task queues (e.g.,
background jobs, retries) |
SQS |
|
Notifications to multiple systems
(e.g., SMS + Email + Lambda) |
SNS |
|
Real-time log processing,
analytics, or ETL pipelines |
Kinesis |
when to Combine Them
These services are often used
together:
- SNS → SQS:
Broadcast messages to multiple SQS queues (fan-out pattern).
- Kinesis + Lambda + SQS: Real-time processing with fallback queues for failed
events.
·
A decision tree and architecture
diagram for choosing between Amazon SQS, SNS, and Kinesis
twtech Insights:
- Here's twtech decision tree and an architecture diagram.
- The decision tree helps twtech to choose between Amazon SQS, SNS, and Kinesis based on its use case.
Decision Tree: Choosing Between SQS, SNS, and Kinesis
┌─────────────────────────────┐ │ Does twtech need real-time-data │ │ Streaming with analytics? │ └──────────────┬──────────────┘ │Yes ▼ ┌──────────────┐ │ Use Kinesis │ └──────────────┘ ▲ │No ┌──────────────┴──────────────┐ │ Does twtech want to deliver the │ │ Same message to multiple │ │ consumers (fan-out)? │ └──────────────┬──────────────┘ │Yes ▼ ┌───────────────┐ │ Use SNS │ │ (fan-out) │ └───────────────┘ ▲ │No ┌──────────────┴──────────────┐ │ Does twtech need message queues │ │ with decoupling, retries, │ │ and delayed processing? │ └──────────────┬──────────────┘ │Yes ▼ ┌───────────────┐ │ Use SQS │ └───────────────┘ ▲ │No ┌─────────┴─────────┐ │ Consider SNS or │ │ another pattern │ └───────────────────┘ Architecture Diagram: Common Usage Scenarios
1. SNS
Fan-Out to SQS Queues
Use case:
- One message sent to SNS is delivered to multiple consumers (via SQS or Lambda, etc.)
2. Kinesis Stream for Real-Time
Processing
Use case:
- High-throughput data ingestion with real-time transformation or analytics.
3. SQS
for Decoupling Microservices
Use case:
- Decouple microservices,
- retry failed jobs,
- buffer traffic.
No comments:
Post a Comment