Query AWS CloudTrail Logs Using CLI

Background

AWS CloudTrail is an AWS service that helps you enable governance, compliance, and operational and risk auditing of your AWS account. Actions taken by a user, role, or an AWS service are recorded as events in CloudTrail.

Visibility into your AWS account activity is a key aspect of security and operational best practices.

You can use CloudTrail to view, search, download, archive, analyze, and respond to account activity across your AWS infrastructure. You can identify who or what took which action, what resources were acted upon, when the event occurred, and other details to help you analyze and respond to activity in your AWS account.

There are many ways to query CloudTrail logs, one can use AWS Athena, third party software, a combination of Glue crawlers and Redshift. In this post I am going to show some practical examples of querying CloudTrail Logs using the AWS CLI.

Query Examples

The CLI has two different commands for querying the logs, the first is by running a query against CloudTrail and the second is by running a query against CloudWatch Logs.

There are some differences in which the commands operate and how many events are brought back. Both commands allow for pagination by providing a token.

See links provided at the end of this post to read up more on the two methods to query events.

Simple Query

aws cloudtrail lookup-events  \
    --start-time $(date -d "-30 minutes" +%s)

aws logs filter-log-events --log-group-name CloudTrail/DefaultLogGroup \
    --start-time $(date -d "-30 minutes" +%s000)

The above queries are simple queries which look back 30 minutes into the logs. Notice the format of the date passed to the –start-time in the two queries. Both of them should output similar content. The output from these queries is not that easy to read so I use ‘jq’ a lot to view the results.

Simple Query with jq

aws cloudtrail lookup-events \
    --start-time $(date -d "-10 minutes" +%s) \
    --query 'Events[].CloudTrailEvent' \
    --output text | jq

aws logs filter-log-events --log-group-name CloudTrail/DefaultLogGroup \
    --start-time $(date -d "-10 minutes" +%s000) \
    --query 'events[].message' \
    --output text | jq

The above queries show only the relevant event information and ‘jq’ helps view the output in a clean manner.

{
  "eventVersion": "1.05",
  "userIdentity": {
    "type": "IAMUser",
    "principalId": "AIDXXXXXXXXXXXXXX",
    "arn": "arn:aws:iam::XXXXXXXXXXXXXX:user/sbali",
    "accountId": "XXXXXXXXXXXXXX",
    "accessKeyId": "XXXXXXXXXXXXXX",
    "userName": "sbali",
    "sessionContext": {
      "sessionIssuer": {},
      "webIdFederationData": {},
      "attributes": {
        "mfaAuthenticated": "true",
        "creationDate": "2020-01-21T06:29:45Z"
      }
    },
    "invokedBy": "signin.amazonaws.com"
  },
  "eventTime": "2020-01-21T07:09:13Z",
  "eventSource": "ec2.amazonaws.com",
  "eventName": "DescribeVolumeStatus",
  "awsRegion": "us-east-1",
  "sourceIPAddress": "X.X.X.X",
  "userAgent": "signin.amazonaws.com",
  "requestParameters": {
    "volumeSet": {},
    "filterSet": {}
  },
  "responseElements": null,
  "requestID": "f2b51353-XXXXXXXXXXXXXX",
  "eventID": "91df27d4-XXXXXXXXXXXXXX",
  "eventType": "AwsApiCall",
  "recipientAccountId": "XXXXXXXXXXXXXX"
}

Lets take a look at another example where we filter and select only certain events.

Query with filter

aws cloudtrail lookup-events \
    --start-time $(date -d "-30 minutes" +%s) \
    --query 'Events[].CloudTrailEvent' \
    --lookup-attributes AttributeKey=EventSource,AttributeValue=kms.amazonaws.com \
    --output text | jq

aws logs filter-log-events --log-group-name CloudTrail/DefaultLogGroup \
    --start-time $(date -d "-30 minutes" +%s000) \
    --filter-pattern '{ $.eventSource = "kms.amazonaws.com" }'  \
    --query 'events[].message' \
    --output text | jq

{
  "eventVersion": "1.05",
  "userIdentity": {
    "type": "IAMUser",
    "principalId": "XXXXXXXXXXXXX",
    "arn": "arn:aws:iam::XXXXXXXXXXXXX:user/sbali",
    "accountId": "XXXXXXXXXXX",
    "accessKeyId": "XXXXXXXXXXXXX",
    "userName": "sbali",
    "sessionContext": {
      "sessionIssuer": {},
      "webIdFederationData": {},
      "attributes": {
        "mfaAuthenticated": "true",
        "creationDate": "2020-01-21T06:29:45Z"
      }
    },
    "invokedBy": "signin.amazonaws.com"
  },
  "eventTime": "2020-01-21T07:09:14Z",
  "eventSource": "kms.amazonaws.com",
  "eventName": "ListAliases",
  "awsRegion": "us-east-1",
  "sourceIPAddress": "X.X.X.X",
  "userAgent": "signin.amazonaws.com",
  "requestParameters": {
    "limit": 1000
  },
  "responseElements": null,
  "requestID": "98376553-XXXXXXXXXXXXX",
  "eventID": "de4c9f8a-XXXXXXXXXXXXX",
  "readOnly": true,
  "eventType": "AwsApiCall",
  "recipientAccountId": "XXXXXXXXXXXXX"
}

In the above example, I added a lookup-attribute for CloudTrail lookup-event query and in the CloudWatch command the filter-pattern was used to limit results.

I prefer the –filter-pattern option as I find it easier to use conditions compared to the –lookup-attributes

aws logs filter-log-events --log-group-name CloudTrail/DefaultLogGroup \
    --start-time $(date -d "-30 minutes" +%s000) \
    --filter-pattern '{ $.eventSource != "kms.amazonaws.com" }'  \
    --query 'events[].message' \
    --output text | jq

{
  "eventVersion": "1.05",
  "userIdentity": {
    "type": "IAMUser",
    "principalId": "XXXXXXXXXXX",
    "arn": "arn:aws:iam::XXXXXXXXXXX:user/tf",
    "accountId": "XXXXXXXXXXX",
    "accessKeyId": "XXXXXXXXXXX",
    "userName": "tf"
  },
  "eventTime": "2020-01-21T07:22:16Z",
  "eventSource": "cloudtrail.amazonaws.com",
  "eventName": "LookupEvents",
  "awsRegion": "us-east-1",
  "sourceIPAddress": "X.X.X.X",
  "userAgent": "aws-cli/1.17.6 Python/3.6.9 Linux/4.15.0-1057-aws botocore/1.14.6",
  "requestParameters": {
    "startTime": "Jan 21, 2020 7:12:15 AM"
  },
  "responseElements": null,
  "requestID": "44b27eca-XXXXXXXXXXX",
  "eventID": "2a1d922d-XXXXXXXXXXX",
  "readOnly": true,
  "eventType": "AwsApiCall",
  "recipientAccountId": "XXXXXXXXXXX"
}

In the above query I used ‘Not Equal to’ expression –filter-pattern ‘{ $.eventSource != “kms.amazonaws.com” }’ to filter out KMS events.

If you prefer to filter or select using ‘jq’ then the following example shows exactly how to do that.

aws logs filter-log-events --log-group-name CloudTrail/DefaultLogGroup \
    --start-time $(date -d "-30 minutes" +%s000) \
    --query 'events[].message' --output text |\
    jq ' . | select(.eventName=="LookupEvents")'

In the above query, I am using ‘jq select’ to filter out events.

Query and List Events

aws logs filter-log-events --log-group-name CloudTrail/DefaultLogGroup \
    --start-time $(date -d "-30 minutes" +%s000) \
    --query 'events[].message' --output text |\
    jq '.eventName' | sort | uniq -c | sort -n

      2 "DeregisterImage"
      2 "DescribeImageAttribute"
      2 "ListAccessKeys"
      2 "LookupEvents"
      3 "DescribeFastSnapshotRestores"
      3 "ListAliases"
      4 "DescribeHosts"
      4 "DescribeKeyPairs"
      4 "DescribePlacementGroups"
      5 "DescribeAccountAttributes"
      5 "DescribeAlarms"
      5 "DescribeClassicLinkInstances"
      5 "DescribeInstanceAttribute"
      5 "DescribeInstanceCreditSpecifications"
      5 "DescribeSecurityGroups"
      5 "DescribeSubnets"
      5 "DescribeVpcs"
      6 "DescribeSnapshotAttribute"
      7 "DescribeAvailabilityZones"
      8 "DescribeLaunchTemplates"
      9 "DescribeLoadBalancers"
     10 "DescribeAddresses"
     10 "DescribeImages"
     14 "DeleteSnapshot"
     14 "DescribeTags"
     15 "DescribeInstanceStatus"
     15 "DescribeInstances"
     15 "DescribeVolumeStatus"
     18 "DescribeSnapshots"
     21 "DescribeVolumes"

The above example show the count by eventName. In this output, we see there were 21 api calls for DescribeVolumes in the last 30 minutes.

Query to find events with error

aws logs filter-log-events --log-group-name CloudTrail/DefaultLogGroup \
    --start-time $(date -d "-30 minutes" +%s000)  \
    --query 'events[].message' \
    --output text |\
    jq ' . | select(.errorCode != null)'

{
  "eventVersion": "1.05",
  "userIdentity": {
    "type": "IAMUser",
    "principalId": "XXXXXXXX",
    "arn": "arn:aws:iam::XXXXXXXX:user/sbali",
    "accountId": "XXXXXXXX",
    "accessKeyId": "XXXXXXXX",
    "userName": "sbali",
    "sessionContext": {
      "sessionIssuer": {},
      "webIdFederationData": {},
      "attributes": {
        "mfaAuthenticated": "true",
        "creationDate": "2020-01-21T06:29:45Z"
      }
    },
    "invokedBy": "signin.amazonaws.com"
  },
  "eventTime": "2020-01-21T07:05:25Z",
  "eventSource": "compute-optimizer.amazonaws.com",
  "eventName": "GetEnrollmentStatus",
  "awsRegion": "us-east-1",
  "sourceIPAddress": "X.X.X.X",
  "userAgent": "signin.amazonaws.com",
  "errorCode": "OptInRequiredException",
  "errorMessage": "An unknown error occurred",
  "requestParameters": null,
  "responseElements": null,
  "requestID": "29306d08-XXXXXXXX",
  "eventID": "b84d4c1b-XXXXXXXX",
  "readOnly": true,
  "eventType": "AwsApiCall",
  "recipientAccountId": "XXXXXXXX"
}

The above query will only pick events if there is an errorCode in the event.

Some can argue that it is better to filter the query itself rather than use ‘jq’, but I find ‘jq’ very useful and more versatile so I prefer using it.

Let me know if you have any questions about this post by commenting below.

Further Reading

Photo Credit

unsplash-logoAlan J. Hendry

1 COMMENT

Leave a Reply