Best Practices for Logging and Monitoring
- Capture log files (obviously) and centralize those log files for systems and applications
- Have the ability to search and analyze the log files. Your log files are useless if you don't know what they mean.
- Implement monitoring and alarming. You want to know about time-sensitive events as they happen.
- Have the ability to trace what applications are doing what through the logs. "This Lambda event caused this EC2 instance to crash which caused X"
- Create dashboards and visualizations for easy viewing.
Amazon CloudTrail
CloudTrail is an AWS service that records activity on the AWS account. It provides history of API and non-API account activity. You can keep logs as long as you want as well as configure it on a per region basis or across all regions. It also offers multiple levels of encryption. For log file integrity validation, it uses SHA-256 hashing and RSA signing. By default, when you enable Amazon CloudTrail, it enables the following:
- Saves 90 days of history
- Logs stored in S3 bucket
- Encrypted using S3 SSE
- Delivery of logs within 15 minutes of API call
Amazon CloudTrail has 3 types of events: Management Events, Data Events, and Insight Events.
Management Events
- When security objects are configured, aka AWS Identity and Access Management (IAM)
- When devices are registered
- When rules are configured for routing
- When setting up logging (I wonder if that means any events related to CloudTrail will be captured? I bet it does!)
Data Events
- S3 activity (Get, Put, Delete). Want to know who's trying to delete your valuable S3 data?
- Lambda function execution activity
- AWS Systems Manager activity
Insight Events
- Unusual API call rate activity (Who in the world is trying to hit my API 10,000 times per second?)
- Unusual error rate activity (Who is trying to login 500 times in the past minute?)
Amazon CloudWatch
CloudWatch allows you to get advanced metrics from your AWS resources. It aggregates the logs, allows you to query, analyze, and create insights and alarms. It can also be used to enable other services like Athena and OpenSearch as discussed below.
Amazon Athena
Amazon Athena can access an S3 bucket which Amazon CloudWatch has put there. From there, it can be queried using standard SQL. It is also serverless, fast, and inexpensive. Compared to OpenSearch, it's also far easier to set up. It also scales automatically.
Amazon OpenSearch
Amazon OpenSearch is an open-source alternative to Athena. Because it's open-source you don't have to pay for a license. It's also very fast and has more features than Athena. Athena also doesn't have visualization built in, but OpenSearch can aid you in visualizing your data right out of the box. A disadvantage is that it requires a bit more set up to get functional as it is required to pull from a Lambda function.
Metrics and Alarming
Using Amazon CloudWatch and AWS EventBridge together, you can create alarms and rules that trigger certain actions. You can do things such as shutting down EC2 instances (Oh no, somebody's hacked it! Shut it down now!), scale instances up and down, and notify the team via email or your favorite chat client. I don't think it supports AOL Instant Messenger (AIM)...bummer.
AWS X-Ray
AWS X-Ray allows you to get a complete view of requests as they travel through your application. It traces through payloads, functions, services, APIs, and more. This isn't necessarily a required service in any infrastructure, but it can be immensely helpful in debugging. I highly recommend reading further in the documentation below. While I don't personally have experience with this (yet!), it will be something I keep in my back pocket.