AWS S3 Event Notifications with SNS, SQS & Lambda | Deep Dive.

AWS S3 Event Notifications with SNS, SQS & Lambda - Deep Dive.

Scope:

• Intro,     
• Core Concepts,
• Understanding the Message Flow,
• Direct Integration Patterns,
• Message Format (SNS → SQS → Lambda),
• Reliability & Delivery Guarantees,
• Handling Idempotency,
• Real-World Best Practices,
• Architecture Variants (options),
• Common Pitfalls (and How to Avoid Them),
• Production Setup Step-by-Step (High Level),
• When to Use What (Quick Decision Matrix),
• Architecture Diagram.

 Intro:

• S3 Event Notifications allow Amazon S3 to publish events like:
• object creation, 
• deletion, 
• lifecycle transitions, 
• restore events, etc. to multiple supported destinations.
• When S3 Event Notifications is integrated with:
• SNS, 
• SQS, 
• Lambda, they form a powerful and decoupled event-driven architecture.

 1. Core Concepts

What S3 Event Notifications Actually Are

• They are push-based triggers fired after S3 processes an event.
• They are best-effort. S3 explicitly does not guarantee delivery, ordering, or exactly-once semantics.
•  twtech configures notifications per bucket and optionally per prefix/suffix.

Supported Destinations

S3 Event Target	Notes
Amazon SNS	Fan-out to multiple subscribers
Amazon SQS	Reliable queueing; decouples producers/consumers
AWS Lambda	Direct compute trigger (sync invocation)
EventBridge	Preferred modern alternative (guaranteed once)

 2. Understanding the Message Flow

Pattern 1: S3 → SNS → SQS → Lambda

• This is a common production pattern for reliability.

[S3 Event] → [SNS Topic] → [SQS Queue(s)] → [Lambda Consumer]

Why this is used:

• SNS provides fan-out to many queues/services.
• SQS provides reliability (retry, DLQ, idempotency control).
• Lambda provides compute.

Flow Characteristics

• Scalable
•  Fan-out to multiple microservices
•  Durable messaging (via SQS)
•   Retryable

 3. Direct Integration Patterns

Pattern 2: S3 → SQS

• Good when you only need durability + batch processing.

Pattern 3: S3 → Lambda

Simple, but:

• No retries from S3 beyond “best effort”
• Risk of event loss
• Not ideal for production-critical workloads

NB:

• Use only when loss is acceptable or events are not critical.

 4. Message Format (SNS → SQS → Lambda)

When SNS publishes an S3 event into SQS, the message looks like:

# json

{

  "Type": "Notification",

  "MessageId": "1111-2222-3333",

  "TopicArn": "arn:aws:sns:us-east-2:accountId:twtechTopic",

  "Message": "{... S3 Event JSON ...}",

  "Timestamp": "2025-01-01T12:00:00Z"

}

NB:

• Inside Message twtech gets the raw S3 event:

# json

{

  "Records": [

    {

      "eventName": "ObjectCreated:Put",

      "s3": {

        "bucket": { "name": "twtech-s3bucket" },

        "object": { "key": "uploads/file1.jpg" }

      }

    }

  ]

}

NB:

• Lambda must parse the nested JSON.

 5. Reliability & Delivery Guarantees

Component	Delivery Guarantee
S3 → SNS/SQS/Lambda	Best effort, not guaranteed
SNS → SQS	Guaranteed at-least-once
SQS → Lambda	At-least-once, with retries and DLQ

NB:

• Ensuring No Event Loss To achieve durable delivery:

S3 → SNS → SQS → Lambda + DLQ

• This is considered AWS best practice for critical workloads.

 6. Handling Idempotency

S3 can send:

• duplicate events
• events with no order
• events for objects that no longer exist
• twtech Lambda should be idempotent:

Techniques:

• DynamoDB deduplication table
• Object version ID checks
• S3 metadata markers
• SQS FIFO queue (if strict ordering is needed)

 7. Real-World Best Practices

A. Always Use SQS in Critical Workloads

• Never rely on S3 → Lambda directly when events are critical.

B. Use SNS Only for Fan-Out

• SNS alone is not durable. Always pair it with SQS.

C. Prefer S3 → EventBridge for Modern Architectures

EventBridge offers:

• guaranteed delivery
• filtering
• schema registry

D. Protect Against "Thundering Herd"

• Large S3 uploads can trigger millions of events.

Use:

• SQS batching
•  Lambda reserved concurrency
•  Consumer scaling policies

E. Use Dead-Letter Queues

SQS → Lambda → DLQ provides visibility into failures.

F. Don’t Forget Prefix/Suffix Filters

These reduce noise significantly:

# json

"Filter": {

  "S3Key": {

    "Rules": [

      {"Name": "prefix", "Value": "incoming/"},

      {"Name": "suffix", "Value": ".csv"}

    ]

  }

}

 8. Architecture Variants (options)

Variant A — Single Queue per Service

S3 → SNS Topic → SQS A → Lambda A

               → SQS B → Lambda B

               → SQS C → Lambda C

NB:

• The above option is Great for Microservices.

Variant B — SQS First-In-First-Out (FIFO) + Ordering

If twtech needs ordering per object prefix:

S3 → SNS FIFO → SQS FIFO → Lambda

NB:

• Only certain S3 event patterns can be guaranteed into the same group.

Variant C — Enriched Workflow

S3 → SNS → SQS → Kinesis → Lambda → DynamoDB

NB:

• Used in analytics workloads.

 9. Common Pitfalls (and How to Avoid Them)

❌   Pitfall: Invisible Event Loss

• Because S3 sends best-effort events, they can simply disappear.

Fix: 

•  Use EventBridge or S3 Inventory for reconciliation.

❌   Pitfall: Duplicate Processing

• S3 can send multiple notifications.

Fix: 

• Deduplication logic in Lambda.

❌   Pitfall: SQS JSON Nested in SNS JSON

• Developers often parse incorrectly.

❌   Pitfall: Queue Not Granted Permission

• S3 and SNS require sqs:SendMessage permissions.
• Make sure the SQS queue policy includes SNS topic ARN.

❌   Pitfall: Lambda Timeout Too Low

When processing large objects, always:

• Use async S3 GETs
• Increase timeout
• Consider streaming

 10. Production Setup Step-by-Step (High Level)

     1.     Create SNS topic s3-events-topic.
2.     Create SQS queue s3-events-queue.
3.     Subscribe the queue to the topic.
4.     Add queue policy giving SNS permission to send messages.
5.     Configure S3 bucket notifications:

# json

{

  "TopicConfigurations": [

    {

      "Id": "PutEvents",

      "TopicArn": "arn:aws:sns:us-east-2:accountID:s3-events-topic",

      "Events": [ "s3:ObjectCreated:*" ],

      "Filter": {

        "Key": {

          "FilterRules": [

            { "Name": "prefix", "Value": "uploads/" }

          ]

        }

      }

    }

  ]

}

     6.     Create Lambda triggered by the SQS queue.
7.     Set DLQ for the SQS queue.
8.     Implement deduplication & error handling in Lambda.

 11. When to Use What (Quick Decision Matrix)

Goal	Best Choice
Reliable delivery	S3 → SNS → SQS
Fan-out	SNS
Simplicity	S3 → Lambda
Strong ordering	FIFO SNS/SQS
Event bus routing	EventBridge
Large-scale ingestion	SQS + Batched Lambda

 12.  Architecture Diagrams