Category: Scalyr

The Essential Guide to Scaling Elasticsearch

Note: This post was originally written for the Scalyr blog. You can check out the original here.

Some things aren’t always what they seem.

You’re tasked with engineering a solution that your organization needs. You implement it with a tool that seems relatively easy to set up. But over time, you realize that there’s no Easy button.

Elasticsearch is an example of one of those things. It’s a great product for collecting event data fairly quickly and easily. You start with one data node in one cluster and go from there. And because it’s free and open-source (for now), it’s even better. But as your Elasticsearch cluster grows and collects more data, you start to have some scaling issues. In this post, I’m going to provide some information on scaling an Elasticsearch implementation, as well as some general recommendations for proactive ways to scale Elasticsearch.

Continue reading “The Essential Guide to Scaling Elasticsearch”

StatsD: How to Measure Anything in Your System

Note: This post was originally written for the Scalyr blog. You can check out the original here.

In his book “How to Measure Anything,” management consultant and author Douglas Hubbard states that “anything can be measured.” Hubbard argues that something that can be observed lends itself to being measured.

How can this apply to software development and operations? Well, in today’s world of increasingly complex IT systems, you can’t afford not to measure anything and everything. But in order to observe and then measure something, it needs to meet the literal definition of observability, meaning that a system’s internal state must be exposed externally. This allows you to measure it.

With observability, you find out not only that your system malfunctioned, but also why. This is done with data from logs, metrics, and traces.

In 2011, the Etsy Engineering team made things a little bit easier to measure and observe metrics in your IT system with the introduction of StatsD. Historically, collecting data about networks and servers has always been easier to do than gaining the same information about applications.

StatsD made collecting application metrics simpler for developers by instrumenting your code with specific metrics you want to observe. As a result, StatsD has become one of the most popular tools for gathering metrics data.

In this post, I’m going to give you a brief tutorial of StatsD and how you can use it to measure anything in your application.

Continue reading “StatsD: How to Measure Anything in Your System”