andyfrench.info

This post follows on from 2 previous posts: Running statsd on Windows and Testing Log4Net.Async with statsd and Graphite. In those posts I set up a virtual Linux box which hosted statsd and Graphite before experimenting with log4Net to compare performance when using different appenders. In this post I’m going to look at some .Net statsd clients to see how they compare with regards to performance and usability.

Now, the performance tests I describe below aren’t particularly scientific given my machine is running a lot of applications - including the virtual machine hosting statsd and Graphite. I ran each of the tests a number of times and the elapsed times I provide below are rough averages and should be taken as indicative of performance only. This is ‘finger in the air’ testing and any conclusions should be taken with a pinch of salt.

Statsd C# Client

I created a really simple console application to send a counter to statsd from inside a ‘for’ loop which ran 100,000 iterations. For the first version I used the Statsd C# Client which I used in my previous posts.

namespace StatsD.Client.Test
{
    using System;

    using StatsdClient;

    /// <summary>
    /// The main program class.
    /// </summary>
    class Program
    {
        /// <summary>The stopwatch.</summary>
        private static System.Diagnostics.Stopwatch stopwatch;

        /// <summary>
        /// Defines the entry point of the application.
        /// </summary>
        /// <param name="args">The arguments.</param>
        static void Main(string[] args)
        {
            Configure();
            Start();

            for (int i = 0; i < 100000; i++)
            {
                Metrics.Counter("test-counter");
            }
            
            Stop();
        }

        /// <summary>
        /// Creates and starts the stopwatch.
        /// </summary>
        private static void Start()
        {
            stopwatch = System.Diagnostics.Stopwatch.StartNew();
        }

        /// <summary>
        /// Stops the stopwatch and outputs the elapsed time.
        /// </summary>
        private static void Stop()
        {
            stopwatch.Stop();
            Console.WriteLine("Elapsed time: {0}", stopwatch.ElapsedMilliseconds);
            Console.ReadKey();
        }

        /// <summary>
        /// Configure th <c>statsd</c> client.
        /// </summary>
        private static void Configure()
        {
            var metricsConfig = new MetricsConfig
            {
                StatsdServerName = "localhost",
                StatsdServerPort = 8125
            };

            Metrics.Configure(metricsConfig);   
        }
    }
}

Running this took 591 milliseconds.

From a usability point of view the API is quite nice offering the static Metrics class used above or a more direct Statsd class which you have to instantiate. The nice thing about the static Metrics class is that you can use it from within a ‘using’ block.

using (Metrics.StartTimer("stat-name"))
{
  DoMagic();
}

However, I’m rather cautious of using static classes as they tend to lead to code that violates the open/closed principle and can make testing awkward. I’m inclined to wrap frameworks in my own service classed that implement interfaces I can easily mock in unit tests.

NStatsD.Client

For the next test I used the NStatsD.Client by Rob Bihun. There haven’t been any commits to the GitHub repo for 7 months or so. This package adds a bit of configuration to the App.config file so no need for configuration in code.

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <configSections>
    <section name="statsD" type="NStatsD.StatsDConfigurationSection, NStatsD.Client" />
  </configSections>
  <startup> 
    <supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.5.1" />
  </startup>
  <statsD>
    <server host="localhost" port="8125" />
  </statsD>
</configuration>

Running the NStatsD.Client version took 1031 milliseconds. Interesting.

The API is ‘no frills’ but gets the job done. The API isn’t using static classes but rather employs the singleton pattern.

NStatsD.Client.Current.Increment("increment");
NStatsD.Client.Current.Increment("increment", 0.5); // Optional Sample Rate included on all methods
NStatsD.Client.Current.Decrement("decrement");
NStatsD.Client.Current.Timing("timing", 2345);
NStatsD.Client.Current.Gauge("gauge", 45);

NStatsD.HighPerformance

Next I tried the NStatsD.HighPerformance package. This is actually a fork of the NStatsD.Client and was created by Aaron Stannard, one of the driving forces behind Akka.Net a really interesting framework for creating distributed systems using the Actor model.

Aaron has a very interesting blog post on the Tradeoffs in High Performance Software which talks about his fork of the NStatsD.Client. The big change is in the private SendToStatsD method of the Client class. Originally it looked like this:

private void SendToStatsD(Dictionary<string, string> sampledData, AsyncCallback callback)
{
	var prefix = Config.Prefix;
	var encoding = new System.Text.ASCIIEncoding();

	foreach (var stat in sampledData.Keys)
	{
		var stringToSend = string.Format("{0}{1}:{2}", prefix, stat, sampledData[stat]);
		var sendData = encoding.GetBytes(stringToSend);
		_client.BeginSend(sendData, sendData.Length, callback, null);
	}
}

By simply changing the async BeginSend call to an inline Send he got an improvement in performance of around 40%.

private void SendToStatsD(Dictionary<string, string> sampledData, AsyncCallback callback)
{
	var prefix = Config.Prefix;
	foreach (var stat in sampledData.Keys)
	{
		var stringToSend = string.Format("{0}{1}:{2}", prefix, stat, sampledData[stat]);
		var sendData = Encoding.ASCII.GetBytes(stringToSend);
		UdpClient.Send(sendData, sendData.Length);
	}
}

There hasn’t been any activity on the GitHub repo for over 9 months so I guess this is considered done.

Running the NStatsD.HighPerformance version resulted in an elapsed time of about 691 milliseconds. Not surprisingly the API is identical to NStatsD.Client.

Graphite

The next package I tried was Graphite, a project inspired by MiniProfiler. This project seems to have a larger scope than some of the others I’ve looked at and provides hooks for Elmah and WCF. With this package you can send metrics to statsd or directly to Graphite (the graphing framework). I note that there haven’t been any commits to the GitHub repo for a year or so.

Running the Graphite version resulted in an elapsed time of about 608 milliseconds.

From an API point of view Graphite looks quite nice. Configuration is done through App.config again and the API provides for ‘using’ statements for timers (you can see the MiniProfiler influence here). For timers there’s a special Step method that basically measure the time taken for the Dispose method to be called.

using (MetricsPipe.Current.Step("duration"))
{
    // Do some work here...
}

Wrapping up

As my old Dad used to say, ‘you pays your money and you takes your choice’. That seems to be the case here. There isn’t a massive difference in performance between any of them. One thing you might like to take in to consideration is the user base.

Judging by the number of NuGet downloads and performance the Statsd C# Client comes out on top.

Package	Elapsed time (ms)	NuGet downloads	Description
Statsd C# Client	591	9830	Allows ‘using’ statements for timers and has static methods for metrics.
NStatsD.Client	1031	1635	Configuration in App.config. Singleton pattern for sending metrics. No ‘using’ statements for timers.
NStatsD.HighPerformance	691	295	Same as NStatsD.Client.
Graphite	608	930	Configuration in App.config. API allows called direct to Graphite. Includes ‘using’ statement for timers.