Amazon S3 Event Notifications | Overview and Hands-On.

Amazon S3 Event Notifications - Overview and Hands-On

Scope:

• Intro,
• Use Cases,
• How It Works,
• Sample json file: S3 Event Notification for Lambda,
• Configuration via UI (console),
• Configuration methods,
• Architecture diagram,
• Permissions for S3 to Invoke the Target,
• Additional Target-Specific IAM Samples,
• Sample SNS topic policy to allow S3 to publish
• Insights,
• Project: Hands-on.

Intro:

• Amazon S3 Event Notifications allow twtech to automatically trigger actions when specific events occur in an S3 bucket. 
• These events could include the creation, deletion, or restoration of an object.

 Use Cases

• Triggering AWS Lambda functions after an object is uploaded.
• Sending messages to an Amazon SQS queue or Amazon SNS topic.
• Starting workflows or automation (e.g., media processing, indexing, or data analysis).

 How It Works

1. Events: twtech defines what events to listen for, such as:
• s3:ObjectCreated:* — any object creation (e.g., upload, copy).
• s3:ObjectRemoved:* — any object deletion.
• s3:ObjectRestore:* — restoring an archived object.
2. Destinations:
• Lambda function – run code automatically in response to the event.
• Amazon SNS topic – send a notification message.
• Amazon SQS queue – queue the message for asynchronous processing.
3. Filters (optional):
• Prefix – only trigger events for objects with a specific prefix (e.g., "images/").
• Suffix – only trigger events for objects with a certain suffix (e.g., ".jpg").

Sample json file: S3 Event Notification for Lambda

# json

{

  "LambdaFunctionConfigurations": [

    {

      "Id": "InvokeLambdaOnImageUpload",

      "LambdaFunctionArn": "arn:aws:lambda:us-east-2:12345678xxx:function:ProcessImage",

      "Events": ["s3:ObjectCreated:*"],

      "Filter": {

        "Key": {

          "FilterRules": [

            { "Name": "prefix", "Value": "uploads/" },

            { "Name": "suffix", "Value": ".jpg" }

          ]

        }

      }

    }

  ]

}

# Configuration methods

• The AWS Management Console (UI)
• The AWS CLI
• Infrastructure as Code tools (e.g., CloudFormation, Terraform)

S3 Event Notifications – IAM Permissions:

• To enable Amazon S3 Event Notifications, twtech must set IAM permissions properly for both:

Architecture diagram:

•      Amazon S3 invokes the target service (like Lambda, SQS, or SNS).
•      twtech user or role configure the S3 event notification.

 1. Permissions for S3 to Invoke the Target

• Depending on twtech target (Lambda, SNS, or SQS), twtech must allow S3 to invoke that resource.

 Example: S3 → Lambda Invocation

Lambda resource-based policy must allow S3 to invoke it:

# json

{

  "Sid": "twtechAllowS3Invoke",

  "Effect": "Allow",

  "Principal": {

    "Service": "s3.amazonaws.com"

  },

  "Action": "lambda:InvokeFunction",

  "Resource": "arn:aws:lambda:us-east-2:accountID:function:ProcessImage",

  "Condition": {

    "ArnLike": {

      "AWS:SourceArn": "arn:aws:s3:::twtech-s3bucket"

    }

  }

}

twtech can attach this using the AWS CLI:

# bash

aws lambda add-permission \

  --function-name ProcessImage \

  --principal s3.amazonaws.com \

  --statement-id AllowS3Invoke \

  --action "lambda:InvokeFunction" \

  --source-arn arn:aws:s3:::twtech-s3bucket

 2. Permissions for the User/Role Configuring the Notification

To allow an IAM user/role to set up S3 event notifications, grant permissions like:

# json 

{

  "Version": "2012-10-17",

  "Statement": [

    {

      "Effect": "Allow",

      "Action": [

        "s3:GetBucketNotification",

        "s3:PutBucketNotification",

        "s3:GetBucketNotificationConfiguration",

        "s3:PutBucketNotificationConfiguration"

      ],

      "Resource": "arn:aws:s3:::twtech-s3bucket"

    }

  ]

}

3. Additional Target-Specific IAM Samples

 S3 → SQS ( Amazon Sinple Queue Service)

SQS queue policy must allow S3 to send messages:

# json

{

  "Effect": "Allow",

  "Principal": { "Service": "s3.amazonaws.com" },

  "Action": "sqs:SendMessage",

  "Resource": "arn:aws:sqs:us-east-1:123456789xxx:MyQueue",

  "Condition": {

    "ArnEquals": { "aws:SourceArn": "arn:aws:s3:::twtech-s3bucket" }

  }

}

 # S3 → SNS (short massage notification)

# Sample SNS topic policy to allow S3 to publish:

# json 

{

  "Effect": "Allow",

  "Principal": { "Service": "s3.amazonaws.com" },

  "Action": "sns:Publish",

  "Resource": "arn:aws:sns:us-east-2:accountID:twtec-Topic",

  "Condition": {

    "ArnLike": { "aws:SourceArn": "arn:aws:s3:::twtech-s3bucket" }

  }

}

 twtech-Insights

• Only one event notification configuration can be set per bucket via the S3 API.

• Permissions must be granted (e.g., S3 must be allowed to invoke twtech Lambda).

Amazon SQS (Simple Queue Service)

Amazon SQS (Simple Queue Service) is a fully managed message queuing service provided by AWS that enables decoupling and scaling of microservices, distributed systems, and serverless applications.

Here are the key concepts and features of Amazon SQS:

 Core Concepts:

1. Queue: A buffer that stores messages until they are processed.
2. Message: A data record stored in a queue (up to 256 KB in size).
3. Producer: Sends messages to the queue.
4. Consumer: Retrieves and processes messages from the queue.

 Types of Queues:

1. Standard Queue (default)
• Nearly unlimited throughput.
• At-least-once delivery (a message might be delivered more than once).
• Best-effort ordering (messages may not be delivered in the order they were sent).
2. FIFO Queue (First-In-First-Out)
• Guarantees exactly-once processing and strict message order.
• Limited throughput (up to 300 messages per second without batching).

 Key Features:

• Scalability: Automatically scales to handle any workload.
• Durability: Messages are redundantly stored across multiple AWS availability zones.
• Visibility Timeout: Prevents other consumers from processing the same message while it's being handled.
• Dead-Letter Queues (DLQs): For messages that fail to process after multiple attempts.
• Long Polling: Reduces cost and latency by waiting until a message is available or timeout.
• Message Delay: Allows delaying the delivery of new messages.

 Use Cases:

• Decoupling microservices
• Asynchronous task processing (e.g., background jobs)
• Buffering and batching workloads
• Integrating with AWS Lambda for serverless architectures.

Project: Hands-on

• twtech creates s3, then configure event notification & EventBridge

twtech Creates an s3-bucket

twtech Creates an s3-bucket: twtech-s3-eventbridge

• twtech configures the event notification in the created bucket: twtech-s3-eventbridge
• twtech Selects the bucket and click open: twtech-s3-eventbridge

• twtech Navigates to the properties tab for the bucket created: twtech-s3-eventbridge

• From properties tab, twtech navigates down to the section : Event notifications, to create event notification.
• Two options come into play: 
• 1, create event notification 

or 

• 2, enable Amazon EventBridge.

Option 2: create event notification 

• twtech enables EventBridge for the bucket: twtech-s3-eventbridge

From: off

To: on

• Remember to: Save changes

Option 2:creates a simple event notification.

• twtech Navigates to the properties tab for the bucket created: twtech-s3-eventbridge

• From properties tab, twtech navigates down to the section Event notifications:  to create event notification.

• twtech Assigns a name: twtech-s3-event-notification

• Event types: specify

Specify at least one event for which twtech wants to receive notifications. 

For each group, twtech can choose an event type for all events, or it can choose one or more individual events.

NB:

• To use the destination: SQS
• twtech needs to Create SQS and allow s3bucket to publish data to: SQS locations.
• Create:  SQS

twtech Searches from the aws services: SQS

• twtech Creates Queue: twtech-sqs-queue

• twtech Assigns a name: twtech-sqs-queue

• twtech enhances the access policy with: Queue policies

From:

• twtech Generates a new policy with: amazon policy generator

• twtech Select policy type: SQS Queue Policy

• twtech Adds statement:
• Principle: *
• Action: SendMessage
• Amazon Resource Name(ARN): arn:aws:sqs:us-east-2:accountID:twtech-sqs-queue

From: SQS policy

• Create Queue: twtech-sqs-queue

• Remember to always add the statement before: generating policy

# Copy the SQS policy generated: SQS policy

# json

{

  "Version": "2012-10-17",

  "Statement": [

    {

      "Sid": "Statement1",

      "Effect": "Allow",

      "Principal": "*",

      "Action": [

        "sqs:SendMessage"

      ],

      "Resource": "arn:aws:sqs:us-east-2:accountID:twtech-sqs-queue"

    }

  ]

• twtech copies Paste in the  Access policy tab: To update the access policy

From:

To:

• twtech Saves changes to: update SQS policy

• twtech Returns to event notification UI: refresh, & select the created sqs

• twtech Saves changes for: event notification

• twtech Goes to SQS Queue & click open to create: twtech-sqs-queue

•  twtech testes event from aws s3 : Poll for messages

From: No messages

To: message sent.

• Select the message and click: To access the message sent

• twtech tests whether the: event notification is working with the SQS.
• twtech Return the bucket and select the bucket configured for event notification (EventBridge), then click open: twtech-s3-eventbridge.

• twtech Uploads the object to the bucket.

• Upload object: Music.mp3

From: in progress 

To: successful

•  twtech verifies that an event message was created after uploading the object: Poll for messeges

• Yes : the message nofication was created.

• twtech Accesses the message to see what exactly happened to the bucket: Notification.
• Message: Michael learns was uploaded to the bucket