Publish CloudTrail logs to CrowdStrike and focus on what matters

Imagine you are running a growing AWS environment. You want to keep tabs on every action, from launching instances to tweaking IAM permissions, so you can stay ahead of potential security incidents. That’s where AWS CloudTrail steps in, your go-to tool for logging user activity and API usage across your accounts. Whether it's keeping your security team informed about suspicious activity or helping you maintain compliance, CloudTrail is essential.

Let's explore with an example. Say a user named “Mary” creates a new IAM user. If Mary isn’t supposed to be creating users, this log entry might point to compromised credentials or overly broad permissions. This is the kind of insight CloudTrail provides, helping you spot and respond to issues before they become real problems.

{
  "eventVersion": "1.08",
  "userIdentity": {
    "type": "IAMUser",
    "principalId": "AIDA6ON6E4XEGITEXAMPLE",
    "arn": "arn:aws:iam::888888888888:user/Mary",
    "accountId": "888888888888",
    "accessKeyId": "AKIAIOSFODNN7EXAMPLE",
    "userName": "Mary",
    "sessionContext": {
      "sessionIssuer": {},
      "webIdFederationData": {},
      "attributes": {
        "creationDate": "2023-07-19T21:11:57Z",
        "mfaAuthenticated": "false"
      }
    }
  },
  "eventTime": "2023-07-19T21:25:09Z",
  "eventSource": "iam.amazonaws.com",
  "eventName": "CreateUser",
  "awsRegion": "us-east-1",
  "sourceIPAddress": "192.0.2.0",
  "userAgent": "aws-cli/2.13.5 Python/3.11.4 Linux/4.14.255-314-253.539.amzn2.x86_64 exec-env/CloudShell exe/x86_64.amzn.2 prompt/off command/iam.create-user",
  "requestParameters": {
    "userName": "Richard"
  },
  "responseElements": {
    "user": {
      "path": "/",
      "arn": "arn:aws:iam::888888888888:user/Richard",
      "userId": "AIDA6ON6E4XEP7EXAMPLE",
      "createDate": "Jul 19, 2023 9:25:09 PM",
      "userName": "Richard"
    }
  },
  "requestID": "2d528c76-329e-410b-9516-EXAMPLE565dc",
  "eventID": "ba0801a1-87ec-4d26-be87-EXAMPLE75bbb",
  "readOnly": false,
  "eventType": "AwsApiCall",
  "managementEvent": true,
  "recipientAccountId": "888888888888",
  "eventCategory": "Management",
  "tlsDetails": {
    "tlsVersion": "TLSv1.2",
    "cipherSuite": "ECDHE-RSA-AES128-GCM-SHA256",
    "clientProvidedHostHeader": "iam.amazonaws.com"
  },
  "sessionCredentialFromConsole": "true"
}

In any active AWS account, there is a constant flow of API actions happening at all times. CloudTrail gathers these actions, batches them into log files, and periodically saves them to S3 as compressed JSON.

Compliance and security frameworks often require CloudTrail logs to be retained and monitored for high-risk events. For example, the CIS Benchmark outlines best practices for securing AWS accounts. However, because CloudTrail logs are stored as batched files on S3, they’re not easy to consume or monitor in real-time. That’s where Streamfold comes in.

With Streamfold’s support for CloudTrail as a data source, we can ingest CloudTrail events in real-time and send them to various destinations. In this article, we’ll focus on delivering CloudTrail events to CrowdStrike’s Falcon LogScale, part of their Next-Gen security information and event management (SIEM) product. With LogScale, we can build queries, create dashboards, and set up real-time alerts based on CloudTrail actions.

Publishing Cloudtrail logs to CrowdStrike Falcon LogScale

Connecting CloudTrail to Streamfold is straightforward and only takes a few steps. Before you begin, create an SQS queue that will be notified whenever a new CloudTrail log file is generated. With Streamfold, there’s no infrastructure to manage, you just provide the details for your CloudTrail S3 bucket, SQS queue, and IAM role.

When you create a new CloudTrail source in Streamfold, it starts in “paused” mode, meaning it won’t consume log files until you switch it to test or active mode. In test mode, the source begins processing SQS log file notifications without removing them from the queue, so notifications will typically reappear after a few minutes.

Next, navigate to Streamfold’s destination page and create a new LogScale destination. If you’re not already a LogScale customer, you can use the Community Edition with free seven day retention. Enter the endpoint for your LogScale account and the ingest token you created.

Finally, create a new stream in Streamfold to connect your CloudTrail source to your LogScale destination. Once it’s created, enable the stream (write to destination) and change your CloudTrail source from Paused to Active mode. If there’s activity in your AWS account, CloudTrail events should start appearing in LogScale within a few minutes.

Brace for the firehose

What you’ll soon notice is that even with an AWS account with reasonably low activity, there’ll be a fairly high volume of events. Many services perform regular API calls to monitor active resources or refresh short-lived security credentials. Not all of these events are useful or required to maintain your compliance and security posture. At large volumes, the number of events makes it harder to detect the critical events and increases the cost of your SIEM. Streamfold makes it easy to control the costs of your SIEM by focusing on the events you care about.

Looking at the event structure of a CloudTrail log, there are two top-level event fields in all events: eventName and eventSource. Using LogScale, let’s take a look at the log volume broken out by the event source.

We’ll run the following query to find the top event sources in total volume:

length := length(@rawstring)
| groupBy(eventSource, function = sum(length))
| sort(_sum)

You can see that STS and EC2 are generating the most log volume. Your own results will likely vary based on what AWS services you’re using.

Now let’s analyze our log volume grouped by the event name.

length := length(@rawstring)
| groupBy(eventName, function = sum(length))
| sort(_sum)

In our case, AssumeRole is at the top of the list, which matches with the results above that listed STS at the top of the list.

We also notice that there are many calls to Describe, List or Get API calls. These are read-only API calls and likely won’t indicate destructive calls to infrastructure. In the next step we’ll remove these read-only events so that you can focus on what matters.

Reduce verbosity, increase signal

As we discovered above, we see a large number of events for read-only API calls. We can add a quick Drop function in Streamfold to immediately exclude these events from getting delivered to CrowdStrike. Here’s what that would look like:

The drop function will be deployed immediately to our pipeline and start filtering events right away. We can check LogScale to see that the total log volume from CloudTrail has dropped off significantly, reducing our costs.

Trimming event bloat

Going back to our top-k list of event names by size, we see that AssumeRole events were contributing a large portion of the total volume. If we look at a single AssumeRole event we get the abbreviated output below:

{
  "eventName": "AssumeRole",
  "eventSource": "sts.amazonaws.com",
  "eventTime": "2024-11-13T22:08:38Z",
  "eventType": "AwsApiCall",
  ...
  "responseElements": {
    ...
    "credentials": {
      "sessionToken": "IQoJb3JpZ2luX2VjEFcaCXVzLWXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXVYw5sbUuQY6wAGabLW1NrRsm3b/tV3gM8HCiLdy7THIcnQRy24383Ticl065vydyTJQoHhcUhOkJPTz3okkZF9NTNvmABqKwZrBPMkMC/FDBQRaBAAhwfv4G3DdWmywlVc3NjnP6XbTj++9YpL+TT/A/bOGgSLcZ9xgXvWrVogjM8n/ChGUdR3X84PItIoS0suPnGURXEFMdxruNtWwNiAKjzzPBoMUruSVNz+60dswF/AvFwzjvNgABX4/YyEfrJw/uWt1c0Wz2/Y="
    }
  }
}

We’ve cut the session token down here for brevity, but before that the token’s full string length was nearly 870 bytes long. This inflates the total event size to over 2kb and the extra session token value field provides no additional value to us.

By adding a remove function it is easy to trim this field and keep the AssumeRole events small. Here’s a screenshot showing how to remove the field, addressed by the path selection responseElements.credentials.sessionToken. Let’s add a filter to this function so that we only operate on AssumeRole events.

With this modification in place, we observe the following additional drop in total log volume:

These are just a couple of examples. There are many other ways you can easily reduce the volume of your CloudTrail events with Streamfold before delivering them to your SIEM. Curating your events with some simple rules can go a long way towards reducing noise in your data and managing your costs.

Closing

CloudTrail logs are essential for strengthening your compliance and security posture. With Streamfold, you can connect your CloudTrail logs to your SIEM, allowing you to focus on important events. By reducing log volumes, you can lower SIEM costs while helping your security operations team detect and mitigate incidents more quickly.

Check out this video, which walks through the stream filtering steps above and shows you how to live sample your CloudTrail logs for quick insights. Sign up for Streamfold today to get the most value out of your CloudTrail logs.