AWS Solutions Architectures Using Lambda, SNS & SQS | Deep Dive.

AWS Solutions Architectures Using Lambda, SNS & SQS - Deep Dive.

Scope:

• Components Overview (Lambda, SNS,SQS),
• Pattern 1: Lambda → SNS → Multiple Subscribers,
• Pattern 1: Lambda → SNS → Multiple Subscribers,
• Pattern 3: Lambda → SNS → SQS → Lambda,
• Common Architecture Patterns,
• Lambda-SNS-SQS Integration Considerations,
• Security Considerations,
• Monitoring & Observability,
• Cost Optimization Tips,
• Sample Architecture Diagram.

1. Components Overview

AWS Lambda

• Serverless compute service.
• Executes code in response to events.
• Key benefits: 
• no server management, 
• automatic scaling, 
• pay-per-use.

Amazon SNS (Simple Notification Service)

• Pub/Sub messaging service.
• Sends messages to multiple subscribers.
• Supports multiple protocols: 
• HTTP/S, 
• Lambda, 
• SQS, 
• email, 
• SMS.

Amazon SQS (Simple Queue Service)

• Fully managed message queue.
• Decouples components of a system.
• Supports two types of queues:
•    Standard: at-least-once delivery, best-effort ordering.
•    FIFO: exactly-once processing, strict ordering.

2. Common Architecture Patterns

Pattern 1: Lambda → SNS → Multiple Subscribers

Use Case: 

• Fan-out to multiple downstream systems.

Flow:

1.      Lambda processes an event (e.g., file upload to S3).
2.      Lambda publishes a message to SNS.
3.      SNS fans out the message to multiple subscribers:

•    Other Lambdas
•    SQS queues
•    HTTP endpoints

Pros:

•  Decouples producer and consumers.
•  Supports multiple simultaneous consumers.
•  Reliable delivery to multiple targets.

Cons / Considerations:

• SNS has a 256 KB payload limit (messages > 256 KB require S3 + pointer).
• Lambda retries on failure might cause duplicate messages (idempotency needed).

Sample Architecture Flow:

User uploads image -> Lambda processes -> SNS topic -> SQS for resizing, Lambda for metadata, email notification

Pattern 2: Lambda → SQS → Lambda (Queue-Based Processing)

Use Case: 

• Asynchronous decoupling and load leveling.

Flow:

1. Lambda produces messages and sends to SQS.
2. SQS stores messages until consumers are ready.
3. Another Lambda triggers on the SQS queue (event source mapping) and processes messages.

Pros:

• Handles spikes in traffic (queue acts as a buffer).
• Smooth scaling of consumers.
• Can configure Dead Letter Queue (DLQ) for failed messages.

Cons / Considerations:

• Message visibility timeout must be tuned to avoid double-processing.
• Max batch size for Lambda triggers (10 messages for SQS FIFO, up to 10,000 for standard with batching).

Sample Architecture Flow:

E-commerce order -> Lambda writes to SQS -> OrderProcessor Lambda consumes -> updates inventory / sends invoice

Pattern 3: Lambda → SNS → SQS → Lambda

Use Case: 

• Fan-out with guaranteed delivery.

Flow:

1.  Lambda publishes message to SNS.
2.  SNS sends message to multiple SQS queues.
3.  Each SQS queue triggers a Lambda for processing.

Pros:

• Combines fan-out (SNS) and buffering (SQS).
• Each consumer can process independently at its own pace.
• Reduces risk of throttling downstream services.

Sample Architecture Flow:

IoT sensor data -> Lambda -> SNS -> SQS1 for analytics, SQS2 for alerting -> respective Lambdas

3. Lambda-SNS-SQS Integration Considerations

Message Size

• SNS max message size: 256 KB.
• SQS max message size: 256 KB.
• For larger payloads, store in S3 and pass reference.

Error Handling

• SQS DLQ: 
• Handles failed messages automatically.
• Lambda retries: 
• Default 3 attempts; 
• combine with DLQ for safety.
• SNS DLQ: 
• For message delivery failures (mainly when subscriber is Lambda or HTTP endpoint).

Idempotency

• Essential for Lambda processing messages from SQS/SNS.
• Use unique IDs to prevent duplicate processing.

Throughput & Scaling

•  Lambda scales automatically but is throttled by:
•    Concurrent execution limits.
•    SQS batch size / inflight messages.
• SQS standard queues: unlimited throughput.
• FIFO queues: limited to 300 TPS without batching.

Ordering Guarantees

• SNS: No ordering guarantees.
• SQS FIFO: preserves order and guarantees exactly-once processing.

4. Security Considerations

        IAM Roles & Policies:

•    Lambda needs permission to publish/consume SNS/SQS.
•    SQS policies to restrict access to specific SNS topics.

        Encryption:

•  Enable KMS encryption for SQS/SNS.

        VPC Integration:

•  Lambdas inside VPC require proper NAT/Gateway for SNS/SQS access.

5. Monitoring & Observability

• CloudWatch Logs: Lambda execution logs.
• CloudWatch Metrics for:
• Lambda: 
• Invocations, 
• errors, 
• Duration.
• SQS: 
• ApproximateNumberOfMessagesDelayed,
• NumberOfMessagesReceived.
• SNS: 
• NumberOfMessagesPublished, 
• NumberOfMessagesDelivered.
• X-Ray: Distributed tracing across Lambda/SNS/SQS.

6. Cost Optimization Tips

• Use SQS batching to reduce Lambda invocations.
• Choose between Standard vs FIFO based on ordering requirements.
• SNS delivery to Lambda is free; delivery to other protocols may cost.
• Retention policies for SQS: keep minimal necessary.

7. Sample Architecture Diagram