Amazon S3 Access Logging | Overview & Hands-On.

 

Amazon S3 Access Logging - Overview & Hands-On.

Scope:

• Intro,
• The concept:  S3 Access Logs,
• How to Enable S3 Access Logging,
• Sample Log Format (Common Log Format),
• Use Cases,
• Considerations,
• S3 Access Logs – Warnings & Caveats,
• Recommended tools to use for integration,
• Project: Hands-on.

Intro:

• Amazon S3 Access Logs provide detailed records for the requests made to twtech S3 bucket. 
• These logs help twtech to track who is accessing its bucket and what operations they are performing (all allowed or denied operations).
• This which is essential for security auditing, performance monitoring, and compliance.
•  Any requests made to twtech S3 bucket, from any account, either authorized or denied, is logged into set (configured) S3 bucket.

The concept:  S3 Access Logs.

S3 Access Logs record details such as:

• Requestor's identity (if known)
• Bucket name
• Request time
• Request action (GET, PUT, DELETE, etc.)
• Response status (200, 403, etc.)
• Object key
• Bytes transferred

NB:

• These logs are delivered to a specified S3 bucket in a log file format.

How twtech Enables S3 Access Logging

1. Go to the S3 Console
2. Select the bucket twtech wants to log (put logs into).
3. Go to the Properties tab.
4. Scroll to Server access logging and click Edit.
5. Check Enable logging.
6. Specify a target bucket (must be in the same AWS region).
7. (Optional) Set a target prefix to organize logs by folder.

 NB:

• The target bucket must not be the same as the source bucket unless lifecycle rules are configured to manage log file growth.

 Sample Log Format (Common Log Format)

# twtech-sample-log

79a4 twtech-EXAMPLE [27/Jun/2025:12:00:00 +0000] 192.0.2.3 arn:aws:iam::12345xxxxx:user/engineer-patpat REST.GET.OBJECT twtech-file.txt "GET /twtech-file.txt HTTP/1.1" 200 - 1024 256 34 27 "-" "aws-sdk-java/1.11.1000 Linux/5.4"

Each log entry contains:

• Bucket owner,
• Bucket name,
• Timestamp,
• Remote IP,
• Requester (if available)
• Operation (e.g., REST.GET.OBJECT)
• Key,
• HTTP status,
• Error code (if any)
• Bytes sent/received,
• User agent.

 twtech Use Cases

• Security & Compliance: Detect unauthorized access attempts.
• Billing Analysis: Track usage by user/account.
• Performance Monitoring: Identify high-frequency access patterns.

 Considerations for S3 logging

• Latency: Logs are delivered with a delay (usually within a few hours).
• Cost: twtech pays for storage of the logs in the destination bucket.
• Permissions: The source bucket must have permission to write to the target bucket:

#  json

{

  "Version": "2012-10-17",

  "Statement": [{

    "Effect": "Allow",

    "Principal": {"Service": "logging.s3.amazonaws.com"},

    "Action": "s3:PutObject",

    "Resource": "arn:aws:s3:::twtech-log-bucket/prefix/*",

    "Condition": {

      "StringEquals": {

        "aws:SourceAccount": "accountID"

      }

    }

  }]

}

twtech-insights:

key warnings and gotchas to be aware of when enabling Amazon S3 Access Logs.

 S3 Access Logs – Warnings & Caveats

1. Potential for Infinite Logging Loop

If twtech enables access logging on the same bucket that stores the logs, every log delivery creates a new log event, causing:

• Recursion: Logs about logs about logs...
• Rapid cost increase
• Storage bloat

✅ Best Practice:
Use a separate logging bucket, or configure lifecycle rules to delete logs after a set time.

2. Delayed Log Delivery

S3 access logs are not real-time. They are delivered:

• With delays ranging from a few minutes to several hours
• Not guaranteed to be delivered immediately or in order

Do not use them for real-time alerting or monitoring.

3. High Volume Buckets Can Generate Tons of Logs

Buckets with high request rates can:

• Generate millions of log entries
• Quickly consume storage space
• Increase costs due to extra storage and access (e.g., Athena or S3 Select)

✅ Use prefixes to organize logs (e.g., by date or bucket).
✅ Set up S3 Lifecycle Policies to archive or delete old logs.

4. Incomplete or Missing Fields

Some fields (like Requester) may be blank if:

• The request is anonymous
• There was an error generating the field
• Access logs are generated before authorization completes

5. Does Not Capture All Access Events

Access logs do not include:

• Events blocked by bucket policies
• Requests blocked before authentication
• Metadata-only access (e.g., HeadObject with 304 Not Modified)

6. Log File Naming is Not Predictable

Log file names are based on:

• Bucket name
• Timestamp
• Random string

 This makes it hard to find specific logs without indexing (e.g., with Athena or Lambda functions).

7. Permissions Must Be Correct

twtech must explicitly grant s3:PutObject to the S3 logging service in the target bucket. If not:

• Logs will silently fail to deliver
• twtech may have no indication something went wrong

Use a policy like:

# json

{

  "Effect": "Allow",

  "Principal": { "Service": "logging.s3.amazonaws.com" },

  "Action": "s3:PutObject",

  "Resource": "arn:aws:s3:::twtech-log-bucket/*"

}

8. Format Is Not Easily Readable

The log format is:

• Dense
• Space-delimited
• Lacks field headers

Recommended tools to use for integration:

• AWS Athena (with predefined schema)
• Amazon QuickSight
• Custom scripts (e.g., Python, jq).

Project: Hands-on

• How twtech creates an s3 bucket and configure it for S3 access logging.
• twtech wants to know who is trying to access it S3 bucket.

twtech Creates s3-logging-bucket: twtech-s3-access-logs

• twtech Assigns a name for bucket: twtech-s3-access-logs

• twtech Enables bucket versioning

• twtech enables the server access logging from another twtech bucket: twtechs3
• Select the bucket and click open : enable logging under bucket properties.

twtech Navigates (scroll down) to: Server access logging and edit to enable

From:

To:

• Specify the target bucket: twtechs3
• Also Select the destination bucket for logs: s3://twtech-s3-access-logs

• twtech saves changes:

• twtech Verifies that the server access logging is: enabled

• twtech verifies that access logging is working for its bucket: twtechs3
• Select the bucket and click open:  to access the objects.

• twtech Accesss (open) the one object in the bucket: Anuwoh FT new.jpg

• Yes: twtech object uploaded is accessed on the browser. 

NB:

• Also, for any new Upload into the target bucket (twtechs3) , logs are sent to the destination bucket (twtech-s3-access-logs)

• twtech Verifies the bucket policy for target bucket twtechs3 : To allows logging from the s3 buckets
• twtech Goes to destination bucket and navigate to: Permissions tab
• Scroll down to : Bucket policy

• twtech Goes to the destination bucket and wait: log will be sent after a couple of minutes.
• twtech verifies that activites in its s3 bucket are successfully logged into: access logging bucket
• Select the destination access logging bucket and click open: twtech-s3-access-logs

• twtech verifies that activites in its s3 bucket are logged into: access logging bucket
• Select the destination access logging bucket and click open: twtech-s3-access-logs

Yes:

NB:

Logs can be used to get an in-depth analysis (Monitoring / observability) of activities associated with the target bucket (twtechs3) access for the destination bucket (access logging bucket): twtech-s3-access-logs

• twtech S3 Activity Logs accessed from browser : UI

twtech - insight: 

S3 Access logs:

• Are time-stamp API calls, 
• indicating what time , 
• someone accessed the target bucket (twtechs3) with twtech ojects, 
• what activity they successful carried out in the bucket,
• what activity they were denied permissions to perform.