How to send SNS Messages to Amazon S3 through Kinesis Data Firehose.

 

To send Amazon SNS messages to Amazon S3 via Kinesis Data Firehose, twtech needs to use AWS Lambda as a bridge, because SNS cannot directly deliver to Kinesis Firehose.

Here's how the architecture and setup work:

 Architecture: SNS → Lambda → Kinesis Data Firehose → S3

# csharp 

[SNS Topic]

     |

     v

[Lambda Function]

     |

     v

[Kinesis Data Firehose]

     |

     v

[Amazon S3 Bucket]

 Step-by-Step Setup

1. Create an S3 Bucket (Firehose destination)

# bash

aws s3api create-bucket --bucket my-sns-logs-bucket

2. Create a Kinesis Data Firehose Delivery Stream

twtech can use the AWS Console or CLI:

# bash

aws firehose create-delivery-stream \

  --delivery-stream-name twtech-sns-firehose-stream \

  --delivery-stream-type DirectPut \

  --s3-destination-configuration '{

    "RoleARN":"arn:aws:iam::<account-id>:role/twtech-firehose-delivery-role",

    "BucketARN":"arn:aws:s3:::twtech-sns-logs-bucket"

  }'

Ensure the Firehose role has PutObject permissions for the S3 bucket.

3. Create a Lambda Function to Process SNS Messages

This function will receive SNS events and send them to Firehose.

Example Lambda (Python):

# python

import boto3

import json

import os

firehose = boto3.client('firehose')

def lambda_handler(event, context):

    for record in event['Records']:

        message = record['Sns']['Message']

        response = firehose.put_record(

            DeliveryStreamName=os.environ['twtech-FIREHOSE_STREAM_NAME'],

            Record={'Data': message + '\n'}

        )

    return {'statusCode': 200}

Add an environment variable:

• FIREHOSE_STREAM_NAME = twtech-FIREHOSE_STREAM_NAME

4. Give Lambda Permissions

• Attach an IAM role to Lambda with:
• sns:Receive
• firehose:PutRecord

5. Subscribe Lambda to SNS Topic

# bash

aws sns subscribe \

  --topic-arn arn:aws:sns:region:account-id:twtech-topic \

  --protocol lambda \

  --notification-endpoint arn:aws:lambda:region:account-id:function:twtech-function

Then grant SNS permission to invoke the Lambda:

# bash

aws lambda add-permission \

  --function-name twtech-function \

  --statement-id sns-invoke \

  --action lambda:InvokeFunction \

  --principal sns.amazonaws.com \

  --source-arn arn:aws:sns:region:account-id:twtech-topic

twtech Security & Best Practices

• Use least privilege IAM roles.
• Enable S3 encryption and Firehose data transformation/compression if needed.
• Implement error handling and DLQ for failed Lambda invocations (via destinations or retry policies).

Kinesis Supported KDF Destinations: