CloudWatch Logs for EC2 Instance | Deep Dive.

Amazon CloudWatch Logs for EC2 Instance - Deep Dive.

Scope:

• Intro,
• The General Setup steps,
• Link to official documentation,
• Key Components,
• Data Flow,
• Sample Agent Configuration (Linux),
• Use Cases,
• Reference Architectures,
• Bottom line.

Intro:

• To monitor twtech EC2 instance using Amazon CloudWatch Logs, it needs to set up an agent on the instance to stream log data to the CloudWatch service. 
• EC2 instances generate system logs, application logs, and OS-level logs. 
• By default, EC2 instance logs live inside the instance (e.g., /var/log/messages, /var/log/syslog, /var/log/httpd/). 
• CloudWatch Logs provides a way to centralize, retain, analyze, and stream logs without manually SSH’ing into instances.

The General Setup steps:

1. Create an IAM Role: Create an AWS Identity and Access Management (IAM) role that grants permissions for EC2 instances to write logs to CloudWatch Logs. The required policy is CloudWatchLogsFullAccess or a custom policy with actions like logs:CreateLogGroup, logs:CreateLogStream, and logs:PutLogEvents. 
2. Attach the Role to Instance: Associate the newly created IAM role with twtech specific EC2 instance.
3. Install the CloudWatch Agent: Connect (SSH) to twtech EC2 instance and install the unified Amazon CloudWatch agent. The agent is available for both Linux and Windows instances.   https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/QuickStartEC2Instance.html
4. Configure the Agent: Create a configuration file (typically in JSON format) that specifies which log files, metrics, and log groups should be monitored and where they should be sent within CloudWatch Logs.
5. Start the Agent: Start the CloudWatch agent using the configuration file you created.
6. Verify Logs in CloudWatch: Navigate to the Amazon CloudWatch console in twtech AWS Management Console under "Logs" > "Log groups" to view the incoming log streams from your EC2 instance.

NB:

• For detailed instructions and to download the necessary agent software, refer to the official Amazon CloudWatch documentation.
•  Or review a specific EC2 setup guide provided by AWS. 

https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/QuickStartEC2Instance.html

 Key Components

1. CloudWatch Agent
• Installed on EC2 to push logs/metrics.
• Replaces the older CloudWatch Logs Agent.
• Supports both logs + metrics in one unified agent.
2. Log Groups & Streams
• Each application/system log file → mapped to a log group.
• Each instance → creates its own log stream within the group.
• Example:
• Log Group: /EC2/twtchAppLogs
• Log Stream: i-1234567890abcdef/httpd-access.log
3. IAM Role / Instance Profile
• EC2 needs permissions (logs:CreateLogGroup, logs:CreateLogStream, logs:PutLogEvents) to send logs.
4. CloudWatch Logs Agent Config
• JSON file defining:
• Which log files to collect.
• Log group and stream names.
• Retention policy.

Data Flow

1. EC2 generates logs (OS, system, application).
2. CloudWatch Agent tails configured log files.
3. Agent batches log events → sends to CloudWatch Logs service.
4. Logs stored in Log Groups/Streams.
5. Optional:
• CloudWatch Logs Insights → search/query logs.
• Subscriptions → stream logs to Lambda, Kinesis, Firehose.
• Metric Filters → turn log patterns into CloudWatch metrics (e.g., count errors).
• Alarms → trigger on those metrics.

# twtech-Sample Agent Configuration (Linux)

{

  "logs": {

    "logs_collected": {

      "files": {

        "collect_list": [

          {

            "file_path": "/var/log/messages",

            "log_group_name": "/EC2/twtechSystemLogs",

            "log_stream_name": "{twtechinstance_id}"

          },

          {

            "file_path": "/var/log/httpd/access.log",

            "log_group_name": "/EC2/HttpdAccess",

            "log_stream_name": "{twtechinstance_id}"

          }

        ]

      }

    }

  }

}

 Use Cases

• Centralized Logging: Collect logs from all EC2s into one place.
• Security & Compliance: Forward logs to SIEM (Security Information & Event Management) or S3 (via Firehose).
• Troubleshooting: Use CloudWatch Logs Insights to query across multiple servers.
• Monitoring: Use metric filters to detect repeated errors, failed logins, or service crashes.

 Reference Architecture (High-Level)

EC2 Instance (generates logs) → CloudWatch Agent (sends logs) → CloudWatch Log Groups (Logs Insights / Subscriptions / Metric Filters) → monitoring,Observability analytics Tools → Streaming  Kinesis Data Streams → Kinesis Firehose (Optional) → Data-lake / S3 Storage → Redshift / OpenSearch (for centralized analytics)

 

Bottom line:

• EC2 + CloudWatch Logs lets twtech to replace SSH + tail -f ....
• EC2 + CloudWatch Logs perfom a centralized, searchable, real-time log streaming, and extend logs to analytics + security platforms.