Reining in Datadog Pricing: How We Reduced Logging Cost with Datadog + Kubernetes
We push a boatload of logs out to Datadog. If you’ve been following our blog, you know that we recently moved all of our hosting to Kubernetes. We have a large Kubernetes cluster hosting 20+ applications, but the major drawback for Datadog logging is pricing.
As we’ve explored hosting solutions, we’ve been documenting some of the challenges we’ve experienced along the way.
- When we were on Heroku, we used Papertrail for log archives. But sadly, we didn’t find it integrated well with Kubernetes. Papertrail is designed around STDOUT logging.
- We tried using fluentd with Elasticsearch and Kibana. But, we found it more complicated to host than we wanted for a logging solution.
- So ultimately, we decided to use Datadog’s log aggregation service, since we also use them for monitoring.
It turns out, our accounting department doesn’t quite appreciate Datadog like we do. So, in order to keep from losing our friend, we knew we had to make it cheaper.
Here’s how we did it.
Datadog’s pricing structure for logging
First, it’s important to understand how Datadog bills for logs.
In our case, we’re running the Datadog agent inside of our Kubernetes cluster using the datadog helm chart. It uses a DaemonSet to deploy a Datadog agent onto each node in the cluster.
Each agent collects all of the logs of the other containers on that node and ships them to Datadog. Upon entering Datadog’s system, the logs are pushed through a series of filters before they are indexed so that they can be searched.
Customers are billed per million indexed events per month. This is a fairly reasonable billing model. But, Kubernetes clusters running a high volume of applications can generate a lot of logs. So for us, this became expensive very quickly.
How to exclude unnecessary logs
It’s fairly easy to exclude logs from Datadog. Simply open the Log configuration menu, select the Index tab, and choose “Add an Exclusion Filter”.
In the dialog, you may enter a Datadog log query just as you would to filter logs in the log search interface. Any logs matching the query will be excluded from the index.
Most importantly: they won’t be counted against your monthly cost.
How to identify logs to exclude
One of the most useful tools for us for defining logging filters was using Datadog’s Log Patterns functionality.
This new tab in your Datadog logging sidebar will quickly give you an idea of the top offenders in your logs. In our case, it helped us to identify that a handful of errors that had little meaning to us were gobbling up a huge percentage of our total log indexing.
We took care of those first, with the idea that we would revisit log usage in a month and see where we were.
Additionally, we had a hard look at which systems were shipping logs to Datadog. We knew we had systems that provided dubious value index logs. We identified that our old staging system was still shipping its logs to Datadog as well, and so we turned that off completely.
Eliminating the noise and reducing the cost
We recently followed these same steps to identify some exclusions that we could make in our own logging configuration.
After about a month, we revisited our log volume and found that we’d nearly eliminated noisy, useless warnings. Even better, we saw a savings of about 18% over previous months’ Datadog bills.
All in all, a pretty good return on investment for less than a half day’s work.
So there it is, spend a little time and save yourself some serious money by indexing fewer logs with Datadog.
At Revelry, we make Platform technology easy to use.