AWS Core Services of Integration & Messaging | SQS, SNS & Kinesis.

Here’s twtech detailed breakdown of AWS Integration & Messaging services — SQS, SNS, and Kinesis covering concepts, key features, benefits, limitations, and common use cases:

AWS Integration & Messaging: SQS, SNS & Kinesis

1. Amazon SQS (Simple Queue Service)

Concept

·        Message Queue service designed to decouple distributed systems and microservices.

·        Stores messages until processed and deleted by consumers.

·        Supports asynchronous, reliable communication between components.

Key Features

·        Two queue types:

o   Standard queues: At-least-once delivery, best-effort ordering, unlimited throughput.

o   FIFO queues: Exactly-once processing, guaranteed order, limited throughput.

·        Visibility timeout: Temporarily hides messages during processing.

·        Dead-letter queues: For handling message failures.

·        Server-side encryption with AWS KMS.

·        Long polling to reduce empty responses and costs.

Benefits

·        Fully managed, scalable, and highly available.

·        Decouples producers and consumers, improving fault tolerance.

·        Supports batching of messages to optimize throughput.

·        Integrates with AWS Lambda for serverless processing.

Limitations

·        FIFO queues have throughput limits (up to 300 TPS without batching).

·        No native support for message filtering or pub/sub model.

·        Messages can be delivered multiple times (Standard queues).

Use Cases

·        Asynchronous task processing (e.g., order processing, image resizing).

·        Buffering requests between components to handle load spikes.

·        Decoupling microservices in event-driven architectures.

2. Amazon SNS (Simple Notification Service)

Concept

·        Fully managed pub/sub messaging service for broadcasting messages to multiple subscribers.

·        Pushes notifications to endpoints (email, SMS, HTTP/S, Lambda, SQS).

Key Features

·        Supports multiple protocols (HTTP/S, Email, SMS, Lambda, SQS, Mobile Push).

·        Message filtering policies to deliver messages selectively.

·        Fan-out architecture by publishing to multiple subscribers simultaneously.

·        Server-side encryption (SSE).

·        Message durability with replication across multiple AZs.

Benefits

·        Enables real-time, push-based messaging.

·        Simplifies fan-out to multiple systems/endpoints.

·        Supports both direct notifications and triggering downstream AWS services.

·        Scales automatically without user management.

Limitations

·        No message persistence (no queueing or retry beyond delivery attempts).

·        No message ordering guarantees.

·        Limited message size (up to 256 KB).

·        Delivery retries follow a fixed policy with limited configurability.

Use Cases

·        Mobile app push notifications.

·        System alerts and monitoring notifications.

·        Fan-out messaging to trigger workflows (e.g., send SMS, invoke Lambda, queue jobs).

·        Broadcasting events to multiple distributed systems.

3. Amazon Kinesis (Data Streams)

Concept

·        Managed service for real-time streaming data ingestion and processing.

·        Allows building data pipelines and streaming applications.

Key Features

·        Sharded data streams provide ordering and parallel processing.

·        Data retention (up to 7 days by default, extendable).

·        Integrates with AWS Lambda, Firehose, and analytics tools.

·        Exactly-once processing with enhanced fan-out.

·        Real-time data ingestion with millisecond latency.

Benefits

·        Enables near real-time analytics and data processing.

·        Scalable and durable streaming infrastructure.

·        Supports complex event processing and aggregation.

·        Integrates well with big data tools and AWS ecosystem.

Limitations

·        Requires manual shard scaling or use of auto-scaling features.

·        More complex to set up and manage compared to SQS/SNS.

·        Higher cost for large scale, continuous streaming.

Use Cases

·        Real-time log and event data ingestion.

·        IoT telemetry and sensor data streaming.

·        Financial transactions and fraud detection pipelines.

·        Machine learning model training with streaming data.

Summary Table

Aspect	Amazon SQS	Amazon SNS	Amazon Kinesis
Messaging Model	Message queue (point-to-point)	Pub/Sub (broadcast)	Streaming data ingestion
Delivery Model	Pull-based	Push-based	Pull-based (sharded streams)
Ordering	FIFO queues support ordering	No ordering guarantees	Ordering within shards
Delivery Guarantees	At least once (standard), Exactly once (FIFO)	Best effort delivery	Exactly-once with enhanced fan-out
Protocols	AWS API	HTTP/S, Email, SMS, Lambda, SQS	AWS API
Retention	4 days max	No retention	Up to 7 days (configurable)
Scalability	Automatic	Automatic	Shard-based scaling
Use Cases	Decoupling, buffering	Notifications, fan-out	Real-time analytics and streaming

twtech Concise Comparison for AWS SQS, SNS, and Kinesis. The core services for integration and messaging on AWS:

Feature	Amazon SQS	Amazon SNS	Amazon Kinesis
Purpose	Message queuing (decoupling components, buffering)	Pub/Sub messaging (broadcasting)	Real-time streaming data processing
Message Model	Point-to-point (queue)	Publish-subscribe (topic)	Data streams (ordered records)
Use Case Examples	Decoupling microservices, buffering tasks, asynchronous processing	Notifications, fan-out to multiple subscribers	Analytics pipelines, real-time metrics, log processing
Message Retention	Up to 14 days	No retention (delivers messages immediately)	Data retention up to 7 days (Kinesis Data Streams)
Delivery Guarantees	At least once delivery, with visibility timeout	Best effort delivery to multiple subscribers	Exactly-once processing (with enhanced fan-out)
Protocol Support	N/A (used via API or SDK)	HTTP/S, Email, SMS, Lambda, SQS, Mobile Push	API, SDK, Kinesis Agents
Scaling	Automatic scaling	Automatic scaling	Scales by shards (manual or automatic via enhanced features)
Ordering	FIFO queues available	No ordering guarantees	Strict ordering within shards
Integration	Can trigger Lambda, used with SNS for fan-out	Pushes to multiple endpoints including SQS queues, Lambda	Integrates with AWS analytics and processing tools like Lambda, Firehose, EMR

twtech Recommendation of When to Use Which

Scenario	Service
Decouple components with reliable, asynchronous message queuing	SQS
Send notifications or fan-out messages to multiple subscribers	SNS
Ingest and process large streams of data in real-time	Kinesis