S3 Access Logging.

 

Amazon S3 Access Logs provide detailed records for the requests made to twtech S3 bucket. 

These logs help twtech to track who is accessing its bucket and what operations they are performing.

This which is essential for security auditing, performance monitoring, and compliance.

 Any requests made to S3 bucket, from any account, either authorized or denied, is logged into another S3 bucket.

The concept:  S3 Access Logs.

S3 Access Logs record details such as:

• Requestor's identity (if known)
• Bucket name
• Request time
• Request action (GET, PUT, DELETE, etc.)
• Response status (200, 403, etc.)
• Object key
• Bytes transferred

These logs are delivered to a specified S3 bucket in a log file format.

How twtech Enables S3 Access Logging

1. Go to the S3 Console
2. Select the bucket you want to log.
3. Go to the Properties tab.
4. Scroll to Server access logging and click Edit.
5. Check Enable logging.
6. Specify a target bucket (must be in the same AWS region).
7. (Optional) Set a target prefix to organize logs by folder.

 The target bucket must not be the same as the source bucket unless lifecycle rules are configured to manage log file growth.

 Sample Log Format (Common Log Format)

# ruby

79a4 twtech-EXAMPLE [27/Jun/2025:12:00:00 +0000] 192.0.2.3 arn:aws:iam::12345xxxxx:user/engineer-patpat REST.GET.OBJECT twtech-file.txt "GET /twtech-file.txt HTTP/1.1" 200 - 1024 256 34 27 "-" "aws-sdk-java/1.11.1000 Linux/5.4"

Each log entry contains:

• Bucket owner
• Bucket name
• Timestamp
• Remote IP
• Requester (if available)
• Operation (e.g., REST.GET.OBJECT)
• Key
• HTTP status
• Error code (if any)
• Bytes sent/received
• User agent

 twtech Use Cases

• Security & Compliance: Detect unauthorized access attempts.
• Billing Analysis: Track usage by user/account.
• Performance Monitoring: Identify high-frequency access patterns.

 Considerations

• Latency: Logs are delivered with a delay (usually within a few hours).
• Cost: twtech pays for storage of the logs in the destination bucket.
• Permissions: The source bucket must have permission to write to the target bucket:

#  json

{

  "Version": "2012-10-17",

  "Statement": [{

    "Effect": "Allow",

    "Principal": {"Service": "logging.s3.amazonaws.com"},

    "Action": "s3:PutObject",

    "Resource": "arn:aws:s3:::twtech-log-bucket/prefix/*",

    "Condition": {

      "StringEquals": {

        "aws:SourceAccount": "123456xxxxxx"

      }

    }

  }]

}

twtech-insights:

key warnings and gotchas to be aware of when enabling Amazon S3 Access Logs.

 S3 Access Logs – Warnings & Caveats

1. Potential for Infinite Logging Loop

If you enable access logging on the same bucket that stores the logs, every log delivery creates a new log event, causing:

• Recursion: Logs about logs about logs...
• Rapid cost increase
• Storage bloat

✅ Best Practice:
Use a separate logging bucket, or configure lifecycle rules to delete logs after a set time.

2. Delayed Log Delivery

S3 access logs are not real-time. They are delivered:

• With delays ranging from a few minutes to several hours
• Not guaranteed to be delivered immediately or in order

Do not use them for real-time alerting or monitoring.

3. High Volume Buckets Can Generate Tons of Logs

Buckets with high request rates can:

• Generate millions of log entries
• Quickly consume storage space
• Increase costs due to extra storage and access (e.g., Athena or S3 Select)

✅ Use prefixes to organize logs (e.g., by date or bucket).
✅ Set up S3 Lifecycle Policies to archive or delete old logs.

4. Incomplete or Missing Fields

Some fields (like Requester) may be blank if:

• The request is anonymous
• There was an error generating the field
• Access logs are generated before authorization completes

5. Does Not Capture All Access Events

Access logs do not include:

• Events blocked by bucket policies
• Requests blocked before authentication
• Metadata-only access (e.g., HeadObject with 304 Not Modified)

6. Log File Naming is Not Predictable

Log file names are based on:

• Bucket name
• Timestamp
• Random string

 This makes it hard to find specific logs without indexing (e.g., with Athena or Lambda functions).

7. Permissions Must Be Correct

You must explicitly grant s3:PutObject to the S3 logging service in the target bucket. If not:

• Logs will silently fail to deliver
• twtech may have no indication something went wrong

Use a policy like:

# json

{

  "Effect": "Allow",

  "Principal": { "Service": "logging.s3.amazonaws.com" },

  "Action": "s3:PutObject",

  "Resource": "arn:aws:s3:::twtech-log-bucket/*"

}

8. Format Is Not Easily Readable

The log format is:

• Dense
• Space-delimited
• Lacks field headers

Recommended tools to use:

• AWS Athena (with predefined schema)
• Amazon QuickSight
• Custom scripts (e.g., Python, jq).

Project: Hands-on

How twtech create s3 bucket and configure access logging.

Create s3-logging-bucket: twtech-s3-access-logs

Assign name for bucket: twtech-s3-access-logs

Enable bucket versioning

How twtech enables server access logging from another twtech bucket: twtechs3

Select the bucket and click open : enable logging under bucket properties.

Navigate (scroll down) to: Server access logging and edit to enable

From:

To:

Specify the target bucket: twtechs3

Also Select the destination bucket for logs: s3://twtech-s3-access-logs

save changes:

Verify whether server access logging is: enabled

How twtech verifies that access logging is working for its bucket: twtechs3

Select the bucket and click open:  to access the objects.

Access (open) the one object in the bucket: Anuwoh FT new.jpg

Yes: The object is accessed on the browser. 

Also, for any new Upload to the bucket twtechs3 (target), logs are sent to the destination bucket(twtech-s3-access-logs)

How twtech Verifies the bucket policy in twtechs3 (target bucket): allows logging for s3 buckets

Go to destination bucket and navigate to: Permissions tab

Scroll down to : Bucket policy

Go to the destination bucket and wait: log will be sent after a couple of minutes.

How twtech verifies that activites in its s3 bucket are logged into: access logging bucket

Select the destination access logging bucket and click open: twtech-s3-access-logs

How twtech verifies that activites in its s3 bucket are logged into: access logging bucket

Select the destination access logging bucket and click open: twtech-s3-access-logs

Yes:

Logs can be used to get an in-depth analysis (observability) of activities associated with the target bucket (twtechs3) access in the destination bucket (access logging bucket): twtech-s3-access-logs

Activity Logs accessed from browser : UI

twtech - insight: 

S3 Access logs:

Are time-stamp API calls, indicating what time , someone accessed the bucket(twtechs3) , what activity they successful carried out in the bucket and what activity they were denied permissions to perform.