Solutions Architectures using Lambda, SNS & SQS | Deep Dive.

A deep dive into AWS solutions architecture using Lambda, SNS, and SQS.

Scope:

•        Design patterns,
•        Best practices,
•        Pitfalls.

Breakdown:

•        Components Overview,
•        Common Architecture Patterns.
•        Lambda-SNS-SQS Integration Considerations,
•        Security Considerations,
•        Monitoring & Observability,
•        Cost Optimization Tips,
•        Sample Architecture Diagram.

1. Components Overview

AWS Lambda

•         Serverless compute service.
•         Executes code in response to events.
•         Key benefits: no server management, automatic scaling, pay-per-use.

Amazon SNS (Simple Notification Service)

•         Pub/Sub messaging service.
•         Sends messages to multiple subscribers.
•        Supports multiple protocols: HTTP/S, Lambda, SQS, email, SMS.

Amazon SQS (Simple Queue Service)

•         Fully managed message queue.
•         Decouples components of a system.
•         Supports two types of queues:
•    Standard: at-least-once delivery, best-effort ordering.
•    FIFO: exactly-once processing, strict ordering.

2. Common Architecture Patterns

Pattern 1: Lambda → SNS → Multiple Subscribers

Use Case: Fan-out to multiple downstream systems.
Flow:

1.      Lambda processes an event (e.g., file upload to S3).
2.      Lambda publishes a message to SNS.
3.      SNS fans out the message to multiple subscribers:

•    Other Lambdas
•    SQS queues
•    HTTP endpoints

Pros:

•         Decouples producer and consumers.
•         Supports multiple simultaneous consumers.
•         Reliable delivery to multiple targets.

Cons / Considerations:

•         SNS has a 256 KB payload limit (messages > 256 KB require S3 + pointer).
•         Lambda retries on failure might cause duplicate messages (idempotency needed).

Example:

User uploads image -> Lambda processes -> SNS topic -> SQS for resizing, Lambda for metadata, email notification

Pattern 2: Lambda → SQS → Lambda (Queue-Based Processing)

Use Case: Asynchronous decoupling and load leveling.
Flow:

1.      Lambda produces messages and sends to SQS.
2.      SQS stores messages until consumers are ready.
3.      Another Lambda triggers on the SQS queue (event source mapping) and processes messages.

Pros:

•         Handles spikes in traffic (queue acts as a buffer).
•         Smooth scaling of consumers.
•         Can configure Dead Letter Queue (DLQ) for failed messages.

Cons / Considerations:

•         Message visibility timeout must be tuned to avoid double-processing.
•         Max batch size for Lambda triggers (10 messages for SQS FIFO, up to 10,000 for standard with batching).

Example:

E-commerce order -> Lambda writes to SQS -> OrderProcessor Lambda consumes -> updates inventory / sends invoice

Pattern 3: Lambda → SNS → SQS → Lambda

Use Case: Fan-out with guaranteed delivery.
Flow:

1.     Lambda publishes message to SNS.
2.      SNS sends message to multiple SQS queues.
3.      Each SQS queue triggers a Lambda for processing.

Pros:

•         Combines fan-out (SNS) and buffering (SQS).
•         Each consumer can process independently at its own pace.
•         Reduces risk of throttling downstream services.

Example:

IoT sensor data -> Lambda -> SNS -> SQS1 for analytics, SQS2 for alerting -> respective Lambdas

3. Lambda-SNS-SQS Integration Considerations

Message Size

•         SNS max message size: 256 KB.
•         SQS max message size: 256 KB.
•         For larger payloads, store in S3 and pass reference.

Error Handling

•         SQS DLQ: Handles failed messages automatically.
•         Lambda retries: Default 3 attempts; combine with DLQ for safety.
•         SNS DLQ: For message delivery failures (mainly when subscriber is Lambda or HTTP endpoint).

Idempotency

•         Essential for Lambda processing messages from SQS/SNS.
•         Use unique IDs to prevent duplicate processing.

Throughput & Scaling

•         Lambda scales automatically but is throttled by:
•    Concurrent execution limits.
•    SQS batch size / inflight messages.
•         SQS standard queues: unlimited throughput.
•         FIFO queues: limited to 300 TPS without batching.

Ordering Guarantees

•         SNS: No ordering guarantees.
•         SQS FIFO: preserves order and guarantees exactly-once processing.

4. Security Considerations

        IAM Roles & Policies:

•    Lambda needs permission to publish/consume SNS/SQS.
•    SQS policies to restrict access to specific SNS topics.

        Encryption:

•  Enable KMS encryption for SQS/SNS.

        VPC Integration:

•    Lambdas inside VPC require proper NAT/Gateway for SNS/SQS access.

5. Monitoring & Observability

•        CloudWatch Logs: Lambda execution logs.
•         CloudWatch Metrics:
•    Lambda: Invocations, errors, duration.
•    SQS: ApproximateNumberOfMessagesDelayed, NumberOfMessagesReceived.
•    SNS: NumberOfMessagesPublished, NumberOfMessagesDelivered.
•         X-Ray: Distributed tracing across Lambda/SNS/SQS.

6. Cost Optimization Tips

•         Use SQS batching to reduce Lambda invocations.
•         Choose between Standard vs FIFO based on ordering requirements.
•         SNS delivery to Lambda is free; delivery to other protocols may cost.
•         Retention policies for SQS: keep minimal necessary.

7. Sample Architecture Diagram