CloudWatch Logs Subscriptions | Overview.

Amazon CloudWatch Logs Subscriptions - Overview.

Scope:  

• Intro,
• Key Concepts,
• Common Uses,
• Architecture,
• vierview: Subscription,
• Key Concepts,
• Destinations,
• IAM Permissions To create a subscription filter,
• Log Filtering Patterns,
• Data Flow Example (Forwarding ERROR logs to Firehose → S3)
• Limits & Quotas,
• Best Practices,
• Subscription vs Export Task,
• Common Pitfalls.

Intro:

• CloudWatch Logs subscriptions provide access to a real-time feed of log events and deliver that feed to other services for:
• Custom processing, 
• Analysis, 
• Or load log events into other systems. 
• This allows for centralized logging and integration with various AWS resources.

Key Concepts 

• 
  Real-time Delivery: Log events matching a specific filter pattern can be sent to a destination as they occur.
• Destinations: Supported destinations for log events include:
• Amazon Kinesis Data Streams
• Amazon Kinesis Data Firehose delivery streams
• AWS Lambda functions
• Filter Patterns: A filter pattern defines which log events are delivered to the destination. If no filter is specified, all log events are sent.
• Scope: Subscription filters can be applied at the log group level (two filters per group) or the account level (using the PutAccountPolicy API). 
• Account-level filters are useful for centralizing logs from many different log groups.
• Permissions: To set up subscriptions, twtech must configure appropriate IAM roles and policies to grant permissions for CloudWatch Logs to send data to the destination and for the destination to process the data. 

Common Uses

• Custom Processing: Use a Lambda function to perform specific logic or transformations on log data.
• Analysis and Alerting: Stream logs to an external service for detailed analysis or to trigger alerts based on specific log content.
• Archival and Long-term Retention: Send logs to Amazon S3 via Kinesis Data Firehose for cost-effective, long-term storage.
• Cross-Account/Cross-Region Sharing: Deliver log data to a centralized logging account or region.

Architecture

1. Ovierview: Subscription

•  A subscription filter is a mechanism that lets twtech to stream log events in near real-time from a CloudWatch Logs group to a destination.
•  Supported destinations:

o   Amazon Kinesis Data Streams

o   Amazon Kinesis Data Firehose

o   AWS Lambda

o   Cross-account destinations (via Kinesis or Firehose, with IAM setup)

NB:

• Unlike Export Tasks (batch jobs), subscriptions are continuous pipelines.

2. Key Concepts

 Log Group

• A subscription is attached to a log group (not an individual stream). 
• Every new log stream inside that group is automatically covered.

 Subscription Filter

        Defines:

•    Pattern: Which log events to forward (can match keywords, numbers, or JSON fields).
•    Destination: Where the logs go.

Example: 

• twtech may choose to forward only ERROR logs to an alerting Lambda.

 Delivery Format

• Logs are delivered as Base64-encoded, gzipped JSON payloads containing batches of events.
• One payload can include up to 1 MB of log events.

3. Destinations

(A) Lambda

•        Logs are invoked as an event payload to twtech Lambda.
•        Useful for:

o   Real-time alerting

o   Metrics extraction (e.g., turning logs into custom metrics)

o   Security checks (filtering sensitive data before persistence)

 NB:

• Must handle decompression and Base64 decoding inside the Lambda.

(B) Kinesis Data Streams

•         Provides real-time stream processing with fine-grained consumers (e.g., Kinesis Analytics, custom apps).

  Pros:

•    Sub-second latency
•    Multiple consumers can process the same logs

        Cons:

• More operational complexity (shards, scaling, checkpoints)

(C) Kinesis Data Firehose

•     Directly delivers logs to S3, OpenSearch, Redshift, or a custom HTTP endpoint.
•      Supports:
•    Compression (GZIP, Snappy, Parquet, ORC)
•    Data transformation (Lambda preprocessing)
•      Most common for CloudWatch Logs → S3 pipelines.

4. IAM Permissions To create a subscription filter:

{

  "Effect": "Allow",

  "Action": [

    "logs:PutSubscriptionFilter",

    "logs:DeleteSubscriptionFilter",

    "logs:DescribeSubscriptionFilters"

  ],

  "Resource": "arn:aws:logs:us-east-2:aacountID:log-group:/twtech/log/group:*"

}

# NB:

On the destination side (e.g., Firehose or Lambda):

o   twtech Must grant logs.amazonaws.com the right to invoke/put records.

# Sample (Lambda permission):

# bash 

aws lambda add-permission \

  --function-name twtech-process-logs \

  --statement-id twtechlogs-id \

  --action "lambda:InvokeFunction" \

  --principal logs.amazonaws.com \

  --source-arn arn:aws:logs:us-east-2:accountID:log-group:/twtech/log/group:*

5. Log Filtering with Patterns

# Match everything:

""

# Keyword:

"ERROR"

# Multiple terms (AND):

"ERROR timeout"

# Numeric filter (if logs structured as JSON):

{ $.status = 500 }

# Wildcard:

?ERROR

6. Data Flow Example (Forwarding ERROR logs to Firehose → S3)

1.  App logs into CloudWatch.

2. Subscription filter on log group:

# bash

aws logs put-subscription-filter \

  --log-group-name /twtech/app/logs \

  --filter-name twtechappErrorToS3 \

  --filter-pattern "ERROR" \

  --destination-arn arn:aws:firehose:us-east-2:accountID:deliverystream/twtech-logs-to-s3bucket

  3. Firehose batches logs → compresses → stores in S3 (twtech-logs-to-s3bucket).

Result: 

• Only ERROR logs end up in twteh S3 bucket.

7. Limits & Quotas

•         One subscription filter per log group (but it can fan out via Kinesis).
•         Delivery retry: up to 24 hours if the destination is unavailable.
•         Event size:

o   Max 256 KB per event

o   Max 1 MB per batch (payload)

8. Best Practices

✅ Centralized pipeline

• Instead of attaching many subscriptions to many log groups, aggregate via Kinesis or Firehose and fan out.

✅ Use Firehose for long-term storage

• Firehose + S3 = scalable, cost-effective archive, easily queried by Athena/Glue.

✅ Use Lambda for enrichment

• Preprocess logs (mask sensitive data, normalize formats) before storage or search.

✅ JSON logs for structured queries

• If logs are JSON, filtering & downstream analytics become much easier.

✅ Cross-account architecture

• Use CloudWatch Destination resource for sending logs across accounts securely.

9. Subscription vs Export Task

Feature	Subscription	Export Task
Latency	Seconds to minutes	Minutes to hours
Destination	Lambda, Kinesis, S3 (via Firehose)	S3 only
Filtering	Yes (patterns)	No
Real-time analytics	✅	❌
Use case	Monitoring, alerts, SIEM	Historical archive

10. Common Pitfalls

•    Forgetting to grant logs.amazonaws.com permission to write/invoke destination.
•    Trying to attach multiple subscriptions directly to one log group (not allowed).
•    Not decompressing Base64+gzip payload in Lambda.
•    Large volume logs → throttling in Kinesis or Lambda → need backpressure handling.