SNS + SQS | Fan-Out Pattern.

 

The SNS + SQS Fan-Out pattern is a classic architecture in AWS for decoupling and distributing messages to multiple services or consumers.

Here's twtech concise breakdown:

The concept: Fan-Out.

Fan-out means publishing a single message to an SNS topic, which then delivers copies of that message to multiple SQS queues (subscribers). Each queue can then be processed independently and in parallel.

 Architecture Overview

# less

    [Publisher]

        |

        v

   [SNS Topic]  --->  [twtech-SQS Queue A] --> [twtech-Consumer A]

        |      

        +------->  [twtech-SQS Queue B] --> [twtech-Consumer B]

        |

        +------->  [twtech-SQS Queue C] --> [twtech-Consumer C]

 How Fan-Out Works

1. Create SNS Topic
2. Create Multiple SQS Queues
3. Subscribe Each SQS Queue to the SNS Topic
• Grant SNS permission to send messages to each queue.
4. Publish to SNS Topic
• The message is delivered to all subscribed queues, creating parallel processing paths.

Benefits of Fan-Out

• Scalability: Consumers can scale independently.
• Decoupling: Publisher doesn’t know or care about who consumes the message.
• Parallel Processing: Each consumer can process the same message differently.
• Fault Isolation: Failure in one consumer doesn’t affect others.

 Security Considerations

• Use IAM policies and SQS access policies to:
• Limit who can publish to the SNS topic.
• Restrict SNS permissions to write to specific SQS queues.
• Enable encryption:
• Use SSE with KMS for both SNS and SQS.
• Private communication:
• Use VPC endpoints (PrivateLink) to avoid traversing the public internet.

 Optional Enhancements

• Dead-Letter Queues (DLQs) for each SQS queue.
• Message Filtering on subscriptions if queues should receive only certain message types.
• Lambda Consumers instead of SQS for event-driven compute.

Example Use Case

• E-commerce order:
• Order placed → SNS publishes → fan-out to:
• Inventory service (SQS A)
• Billing service (SQS B)
• Shipping service (SQS C)