Enhance your Netlify log drains with Streamfold

Mike Heffner

September 18, 2024

·Product

Here at Streamfold we're enthusiastic users of the Netlify Platform. Some of us have been deploying sites to Netlify for years and others even worked there in the past! So when it came time to decide where to host our marketing and application sites, Netlify was an obvious front runner. With Netlify, we can deploy confidently without worrying about scaling or availability challenges.

Netlify and Streamfold

So today we're excited to announce the release of our Netlify log drains integration. With this integration you get complete visibility into your Netlify site, including:

request traffic logs
functions runtime
edge functions runtime
build deploys

On the Netlify enterprise plan you have unlimited log drain access at no additional charges. When you combine your Netlify logs with Streamfold, you gain complete control over your telemetry data. This means you can route logs to multiple destinations, protect user's PII data, easily transform data with programmatic functions, and so much more. Let’s dive into some of the powerful capabilities Streamfold unlocks.

What is Streamfold?

Before diving in, here's a quick introduction to Streamfold:

Streamfold is a real-time telemetry pipeline designed for developers to give you full control over your logs, metrics, traces, and events. You can configure it on the fly to route your telemetry data to various destinations, filter out noise, or enforce specific data formats. With real-time metrics and live event sampling, Streamfold gives you the insights you need to manage your data safely and effectively.

A stream with a Netlify source

Netlify + Streamfold

So, what can you do when you connect your Netlify log drains to Streamfold? Here are just a few examples of what Streamfold can provide. We plan to dive into these further in future posts as well.

Support for Open Telemetry

OpenTelemetry is the emerging standard for telemetry data exchange, covering metrics, logs, and traces. By connecting Netlify to Streamfold, you can convert your Netlify logs into OpenTelemetry and send them to any OpenTelemetry-compatible observability backends.

Streamfold even has built-in support for the Honeycomb Observability Platform. With just a few clicks, you can start sending your Netlify logs to Honeycomb to gain complete visibility into your site's performance and traffic. You can then use Honeycomb’s query builder to dig deep into your data, explore performance trends, and set up dashboards that keep your team informed.

Honeycomb Query

In a future post we'll explore how to track invocations to your Netlify Functions and correlate that with the performance of your backend API.

Protect PII

Netlify logs may contain Personally Identifiable Information (PII) that shouldn't be exposed in downstream systems. However, rather than omitting this data from the log drain, you can use Streamfold to mask it so that you can still identify unique user personalities without risking their PII.

For example, you can hash IP addresses to maintain uniqueness without revealing the actual IP. Here’s a quick example using a custom Bloblang code function in Streamfold:

root = this
root.client_ip_hash = root.client_ip.hash("hmac_sha1", "secret-key").encode("hex")
root.client_ip = deleted()

Our docs contain more examples on how to anonymize and encrypt PII data.

Archive to S3

With data compliance regulations and the need for historical forensics, keeping log data for long periods is often necessary. Cloud object storage, like Amazon S3, provides a cost-effective means to archive data for long term retention. With Streamfold you can archive raw data to S3 while routing important operational data to your observability tools. Data in S3 can be easily queried with Amazon Athena or imported into customer warehouses for further analysis.

Rich analytics dashboards

Netlify traffic logs are invaluable for understanding your site's performance and customer behavior. Unlike browser based analytics solutions that suffer from ad-blocking blind spots, server-side logs give you full visibility. By routing your Netlify logs to tools like PostHog, you can create rich analytics dashboards that your whole team can use.

We’ll cover how to gain further analytics insight into your Netlify site in a future post.

Let's Get Started!

If you're a Netlify customer, and you would like to enhance your log drains with Streamfold, give us a shout. We're currently in private beta and can have you up and running in a few minutes.

Once you're onboard, connecting your Netlify log drains to Streamfold only takes a few steps:

Head to your Streamfold Add Source page and select Netlify
Enter a name for your new source and hit save.
In Netlify, head to the log drains configuration and add a General HTTP source. Follow the instructions on the Streamfold page to fill out the right options.

Netlify Configuration Instructions

NOTE: Netlify log drains are only supported on the Netlify Enterprise plan.