Amazon SQS FIFO Queue (First-In-First-Out) - Overview & Hands-On.
Scope:
- Intro,
- Key Features of SQS FIFO Queues,
- Key Configuration Parameters,
- Use Cases,
- FIFO vs Standard Queue Comparison,
- Security & Monitoring,
- Project: Hands-on.
Intro:
Amazon SQS FIFO (First-In-First-Out)
queues are designed to ensure exactly-once processing and message
order preservation.
Here's twtech Overview of how Amazon SQS FIFO (First-In-First-Out) queues work and what makes them different from standard queues:
Key Features of SQS
FIFO Queues
|
Feature |
Description |
|
Order Preservation |
Ensures messages are processed in
the exact order they are sent (within a message group). |
|
Exactly-Once Processing |
Prevents duplicate message
delivery using deduplication mechanisms. |
|
Message Groups |
Messages with the same MessageGroupId
are always processed in order; multiple groups can be processed in parallel. |
|
Limited Throughput |
300 transactions per second (TPS)
by default (can be increased with batching). |
|
Supports Message Batching |
Send or receive up to 10 messages
in a single batch to improve throughput. |
|
Deduplication |
Ensures no duplicate messages are
delivered. Deduplication is based on content or a MessageDeduplicationId. |
Key Configuration
Parameters
|
Parameter |
Purpose |
|
Content-Based Deduplication |
If enabled, SQS uses a SHA-256
hash of the message body to deduplicate. |
|
MessageGroupId |
Required for FIFO queues. Groups
related messages for ordered processing. |
|
MessageDeduplicationId |
Optional unless content-based
deduplication is disabled. Unique ID for deduplication within 5 minutes. |
twtech Use Cases.
- Financial transactions – where order matters.
- Inventory management
– maintain consistent state.
- Workflow systems
– steps that must happen in sequence.
FIFO vs Standard Queue
Comparison
|
Feature |
FIFO Queue |
Standard Queue |
|
Ordering |
Strict (per message group) |
Best-effort (not guaranteed) |
|
Duplicates |
Eliminated |
Possible |
|
Throughput |
Limited (300 TPS/group) |
Nearly unlimited |
|
Use Case |
Order-sensitive workflows |
High-volume, unordered data |
Security &
Monitoring
- IAM policies
for access control.
- Server-side encryption (SSE) available with AWS KMS.
- CloudWatch metrics
for monitoring and troubleshooting.
- Dead Letter Queues (DLQs) to handle message processing failures.
Project: Hands-on
- How twtech creates an SQS(simple queue service) and
configure for FIFO(First-In-First-Out
- Search for the AWS service: SQS
- Create a queue and assign a name: twtech.fifo
NB:
- The name must end with .fifo for it to be created.
Configuration
- Set the maximum message size, visibility to other consumers, and message retention.
Deduplication scope
- Specify the scope of deduplication for a FIFO queue.
Encryption
- Amazon SQS provides in-transit encryption by default.
- To add at-rest encryption to twtech queue, enable server-side encryption.
Access policy
- Define who can access twtech queue.
- Create Quequ: twtech.fifo
- twtech send and receive messages from fifo queue.
- Configure then send message with Message group ID: twtech-fifo-id-1
- Message deduplication ID: twtech-dedu-id-1
- Send the message: Hello twtech fifo team 1
- Send another message for twtech team 2 : Hello
twtech fifo team 2
- Configure then send message with Message group ID: twtech-fifo-id-2
- Message deduplication ID: twtech-dedu-id-2
- Send message: Hello twtech fifo team 2
- Send another message for twtech team 3: Hello
twtech fifo team 3
- Configure then send message with Message group ID: twtech-fifo-id-3
- Message deduplication ID: twtech-dedu-id-3
- Send message: Hello twtech fifo team 3
- Send another message for twtech team 3: Hello
twtech fifo team 4
- Configure then send message with Message group ID: twtech-fifo-id-4
- Message deduplication ID: twtech-dedu-id-4
- How twtech accesses all the four messages sent to twtech-teams (1.2,3,4): by clicking on each message ID.
- Message to: twtech-team 1
- Message
to: twtech-team 2
- Message
to: twtech-team 3
No comments:
Post a Comment