Saturday, 4 April 2015

Running statsd on Windows

We are going to be using statsd (or possibly Heka) to consume and analyse performance metrics for our applications. Being a curious soul I wanted to start playing with statsd on a Windows 8.1 machine at home but quickly discovered that the best approach is to go with Linux.

There are quite a few projects knocking around on GitHub to help build virtual machines with statsd and Graphite – a graphing solution that allows time series data to be visualised – but I chose José Padilla’s statsd-graphite-vm as a starting point although I had to customise things a little because I didn’t have all the necessary infrastructure in place.

José’s solution makes use of several technologies:

  • VirtualBox - a x86 and AMD64/Intel64 virtualization product
  • Vagrant – orchestrates the provisioning of the virtual environment
  • Chef – an automation platform for building virtualised environments

 

At the end of the process I ended up with a virtual Linux box running 32-bit Ubuntu complete with statsd and Graphite.

Cloning the statsd-graphite-vm project

The first step was to clone the statsd-graphite-vm project from GitHub. I chose to use SourceTree because it’s nice.

image

Installing VirtualBox

OK, I love Chocolatey. I was late to the game here but I have to say if you’re not using it already you really should. There’s no messing around finding the latest version of something or worrying about 32 or 64-bit distributions. You just ask Chocolatey to install a package by name and away it goes.

So, the next thing to do here is to install VirtualBox. Time to fire up Chocolatey.

image

That’s it. VitualBox is installed. VirtualBox is described in the following terms:

“VirtualBox is a powerful x86 and AMD64/Intel64 virtualization product for enterprise as well as home use. Not only is VirtualBox an extremely feature rich, high performance product for enterprise customers, it is also the only professional solution that is freely available as Open Source Software under the terms of the GNU General Public License (GPL) version 2.” - https://www.virtualbox.org/

Basically, VirtualBox gives us a virtual machine into which we can load an operating system etc. a process which is greatly simplified by using Vagrant. 

Installing Vagrant

The next piece of the puzzle is Vagrant. Chocolatey to the rescue again.

image

So what part is Vagrant playing? Vagrant is described thus:

“Vagrant provides easy to configure, reproducible, and portable work environments built on top of industry-standard technology and controlled by a single consistent workflow to help maximize the productivity and flexibility of you and your team.

To achieve its magic, Vagrant stands on the shoulders of giants. Machines are provisioned on top of VirtualBox, VMware, AWS, or any other provider. Then, industry-standard provisioning tools such as shell scripts, Chef, or Puppet, can be used to automatically install and configure software on the machine.” - https://docs.vagrantup.com/v2/why-vagrant/

And:

“If you're a developer, Vagrant will isolate dependencies and their configuration within a single disposable, consistent environment, without sacrificing any of the tools you're used to working with (editors, browsers, debuggers, etc.). Once you or someone else creates a single Vagrantfile, you just need to vagrant up and everything is installed and configured for you to work.” - https://docs.vagrantup.com/v2/why-vagrant/

If you look inside the statsd-graphite-vm folder you’ll find a Vagrantfile. This file is read by Vagrant and is used to define the type of virtual machine we want including instructions to run Chef to setup software etc. Chef is being used to create a ‘recipe’ of all the ingredients we need included in the new virtual machine.

My customisation - Getting the Chef cookbooks

Now, the statsd-graphite-vm solution uses Chef to configure all the bits and pieces that go in to building the virtual machine. In fact, the statsd-graphite-vm project suggests you use librarian-chef to automatically download all the Chef cookbooks required to build the VM by running ‘librarian-chef install’ as the first step in the process of provisioning the new virtual machine. This really simplifies things – or at least it should - but after a lot of time spent installing Ruby and getting librarian-chef up-and-running I was still having problems (e.g. SSL certificate issues).

Life’s too short so I abandoned that approach and elected to download the cookbooks manually. To do that I created a ‘cookbooks’ folder under statsd-graphite-vm, hit the interweb and Googled with Bing to find the necessary cookbooks and put them in the cookbooks folder myself. A bit of a pain but it worked.

If you choose to do the same the full set of cookbooks looks like this:

image

If you’re looking for Chef cookbooks a great place to start is the Chef Supermarket. An alternative for some of the cookbooks used here is Hector Castro’s contributions on GitHub. I used a variety of locations which I’ve listed here:

 

Firing up the VM

At this point things are just about ready to go but I found that I needed to change a port number in the Vagrantfile to avoid a conflict. I changed the forward_port for port 80 to be 8888 from the default 8080.

# -*- mode: ruby -*-
# vi: set ft=ruby :
#^syntax detection

VAGRANTFILE_API_VERSION = "2"

Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|

  # Every Vagrant virtual environment requires a box to build off of.
  config.vm.box = 'precise32'
  config.vm.box_url = 'http://files.vagrantup.com/precise32.box'

  # Forward a port from the guest to the host, which allows for outside
  # computers to access the VM, whereas host only networking does not.
  config.vm.network 'forwarded_port', guest: 80, host: 8888
  config.vm.network 'forwarded_port', guest: 8125, host: 8125, protocol: 'udp'
  config.vm.network 'forwarded_port', guest: 8126, host: 8126

  config.vm.provision 'chef_solo' do |chef|
    chef.add_recipe 'apt'
    chef.add_recipe 'graphite'
    chef.add_recipe 'nodejs'
    chef.add_recipe 'statsd'
  end

end

You can also see in the Vagrantfile that the build is based on a ‘box’ called precise32 (in this case a 32-bit Ubuntu 12.04 build). What is a box?

“Instead of building a virtual machine from scratch, which would be a slow and tedious process, Vagrant uses a base image to quickly clone a virtual machine. These base images are known as boxes in Vagrant, and specifying the box to use for your Vagrant environment is always the first step after creating a new Vagrantfile.” - http://docs.vagrantup.com/v2/getting-started/boxes.html

Precise32 is a standard Ubuntu 12.04 LTS 32-bit box from HashiCorp's Atlas box catalog.

Once all the above had been done I was able to start the VM by issuing a simple Vagrant command. I opened a command prompt and changed to the statsd-graphite-vm folder before running the following:

vagrant up
vagrant ssh

And we are in!

image

So, let’s see what we get if we browse http://localhost:8888 in a web browser.

image

Boom shakalaka, rude boy!