Back 
In AWS, there are two ways to log access to S3 storage resources, i.e. buckets and bucket objects:
- server access logging (https://docs.aws.amazon.com/AmazonS3/latest/dev/ServerLogs.html)
- bucket object-level logging using CloudTrail (https://docs.aws.amazon.com/AmazonS3/latest/user-guide/enable-cloudtrail-events.html)
However, understanding the differences and how to configure each one can be confusing and complex. In this blog, we’ll explore the functionality and caveats of both and why you would want to use one versus the other.
Server Access Logging
Server Access Logging is similar to http server logging in the kind of information logged. It answers the general question, “Who is making what type of access to which objects?” Server access logging has several limitations that make it a non-starter for production IT/security needs, but is straight-forward to understand and configure.
Figure 1. Server Access Logging Architecture
In the flow above, a PDF file is written to the bucket with a CLI or API command. Server Access Logging generates one line per bucket operation and writes it to a text log file that is uploaded to an S3 log bucket that you specify when you configure server access logging. The logged information is from the perspective of the “server,” which you can associate with the public REST API endpoint on the AWS side. The formatting is loosely-structured with a known ordering of fields, space-delimited, with quote escaping for fields containing spaces. You can use S3 features in the logging bucket to configure data retention for the logs and must build your own notification system for events of interest or to analyze the log file records.
Logged Operations
Server access logging provides records of the requests that are made to a bucket. Each access is recorded as one text record with the exception of a copy (which is recorded as a delete and write). Each log entry has 24 fields that fall into 4 general categories:
- HTTP/REST operation (e.g. GET, PUT, POST, OPTIONS, etc.)
- Requester information (including user agent, AWS account, IP)
- Resource (bucket or bucket object)
- Session information (such as data size, response times, authentication type).
The information logged summarizes the access operation but does not necessarily provide full payload details. For example, a server access log entry for a PUT ACL operation on an object does not include the new ACL definition.
Example
Here is an example entry with some of the more useful fields highlighted:
Figure 2. Server Access Logging Entry
Fields
The data fields are described in the AWS documentation (https://docs.aws.amazon.com/AmazonS3/latest/dev/LogFormat.html) and summarized in this table along with example requests:
Since the fields are space delimited with double-quote escapes, a common way to parse these records for analysis or eventing will be to use regular expression parsing. In order not to mis-parse the data, care must be taken to accommodate the range and type of values for each field since fields have been added as recently as the past 12 months.
Note as well that S3 server access logging is a bucket-level configuration, meaning that certain bucket operations will not be captured such as bucket listings (which is against the S3 service itself) and bucket creation and deletion. This is in contrast to S3 Object-Level Logging with CloudTrail which can record all API calls regardless of the state of any individual bucket.
Configuration
- The AWS Console will show the Server Access Logging option during bucket provisioning/creation:
Figure 3. Server Access Configuration During Bucket Creation
- The server access logging setting is fully accessible from the AWS CLI and REST API.
- The Log Delivery group is used by the server access logging facility when writing to the S3 Log Bucket, so the S3 Log Bucket ACL should grant the Log Delivery group write access.
Object-Level Logging (CloudTrail)
Object-Level Logging, sometimes referred to as S3 CloudTrail logging, saves events in json format in CloudTrail, which is AWS’s API-call eventing service. Once in CloudTrail, detailed events are stored in an S3 Bucket, and can be easily integrated with other services such as CloudWatch (monitoring/alerts), SNS (notifications), SQS (queues for other processing), and lambda functions (serverless processing). Object-Level Logging is more complicated to understand and configure and has some additional costs, but provides advanced functionality to address all logging use cases.
Figure 4. Object-Level Logging (CloudTrail) Architecture
In the flow above, you can see that Object-Level logging involves more services than server access logging, specifically:
- CloudTrail (for recording API call events) and typically,
- CloudWatch (for notifications, alarms, and metrics)
When any bucket operation is performed, a more detailed and structured event (json format) is generated in CloudTrail, which is configured to store the event data in an S3 Log bucket. For notifications, CloudWatch is typically used as it has rich filtering functionality for matching specific events and can generate metrics with alarms and notifications targeting SNS, SQS, or lambda functions. Retention has to be configured both in CloudWatch as well as the S3 Log Bucket.
Logged Operations
All bucket API calls, both at the bucket level and object level, can be logged, including:
- Buckets: GET,PUT,DELETE on bucket, cors, encryption, lifecycle, policy, replication, tagging, versioning, etc.
- Objects: GET,DELETE,PUT,POST,HEAD on object, acl, tagging, part, multipart, torrent, etc.
In contrast to server access logging, you can see full details about operations such as PUT ACL on an object, including the ACL definition.
The list of APIs that are logged are documented in the CloudTrail documentation: (https://docs.aws.amazon.com/AmazonS3/latest/dev/cloudtrail-logging.html), but note that is out-of-date and lags behind the actual APIs offered. A detailed table follows later in the Comparison section that includes a complete API list and compares which operations are logged with object-level logging vs. server access logging.
Example
Any access to the bucket or bucket object is logged in a detailed json data structure on an S3 bucket of your choosing. Here is an example entry: