The life and times of Jeffrey Forman.

The Internet is slow. Is the Internet down?

We have all heard the same questions at one point in our careers, “Is the Internet down?” or “Getting to X site is slow.” You scramble to a browser to see if Google, ESPN or the NY Times websites are up. Then you fire up traceroute. In some cases, the pages might load slowly, in other cases not at all. These two situations are often downstream fallout of two connectivity issues: latency and packet loss. Latency is the time it takes for a packet to get from source to destination. The speed of light says the latency for one packet to get across the USA from New York to San Francisco is normally between 70-90ms [1]. Packet loss occurs when packets do not make it from their source to destination, being lost along the way. Many factors can contribute to packet loss, including overloaded routers and switches, service interruptions, and human error.

When diagnosing network issues between source and destination, it is helpful to have data to backup your suspicions of slow and inconsistent network performance. Insert Smokeping.

As part of a network and system monitoring arsenal, you might have Nagios configured for host and service monitoring, and Munin for graphing system metrics. But for monitoring network performance, I feel Smokeping fills that gap. Below are some notes I took getting Smokeping installed and running on a Ubuntu Linux VM at home.

I installed Smokeping from source, since the version in the Ubuntu repository (2.3.6 for Ubuntu Oneiric) is quite old compared to the latest release, 2.6.8, at the time of this post. After installing the various dependencies from the Ubuntu repo, I was able to build and install Smokeping under /opt/smokeping. One thing I do appreciate about Smokeping is that you can run it as any arbitrary user. No root needed!

First we need to configure Smokeping and verify it starts up.

Part of my Smokeping config:

 imgcache = /opt/smokeping/htdocs/cache
 imgurl   = http://yourserver/smokeping/cache

+ random
 menu = random
 title = Random Hosts

++ utexas
 host = http://www.utexas.edu

++ stanford
 host = http://www.stanford.edu

++ mit
 host = media.mit.edu

++ multihost
 title = edu comparison
 host = /random/utexas /random/stanford /random/mit
 menu = EDU Host Comparison
 

imgcache must be the absolute path on your webserver where Smokeping’s cgi process writes out png files. imgurl is the absolute URL where your httpd presents the imgcache directory.

What follows is a sample stanza under the ‘charts’ category in the config. It contains three discrete Smokeping graphs to webservers found at the MIT’s Media Lab, University of Texas, and Stanford University. I picked these three hosts because they represent a variety of near, far, and trans-continental servers from my home in the Northeastern US. The last entry, multihost, creates one single graph with the three data points combined. The ‘host’ parameter in this case contains three path-like references to the graphs we want consolidated into one graph.

To test that Smokeping starts up, execute the following:

jforman@testserver1 /opt/smokeping % ./bin/smokeping --config /opt/smokeping/etc/config --nodaemon
Smokeping version 2.006008 successfully launched.
Not entering multiprocess mode for just a single probe.
FPing: probing 3 targets with step 300 s and offset 161 s.

When you are ready to take the training wheels off, remove the ‘–nodaemon’ argument, and put this command in your distribution’s rc.local file to be started at boot time.

To actually view the generated data in graphs, you will need CGI support configured in your httpd of choice. For the most part, I run Apache.

Snippets of required Apache configuration:

LoadModule cgi_module /usr/lib/apache2/modules/mod_cgi.so
AddHandler cgi-script .cgi

Alias /smokeping "/opt/smokeping/htdocs"
<Directory "/opt/smokeping/htdocs">
  Options Indexes MultiViews
  AllowOverride None
  Order deny,allow
  Allow from all
  Options ExecCGI
  DirectoryIndex smokeping.cgi
</Directory>

I am not presenting my Smokeping install as a virtual host, so I have left that part out. Also take note that the httpd’s user needs to have permissions on the imgcache directory in your Smokeping config file. In my case, /opt/smokeping/htdocs/cache is 775 with www-data as the group.

Once things have been running for even a short as 30 minutes, you will begin to see data. An example of the the last three hours performance graph to MIT’s media lab:

Here we see roughly a 10ms trip time from my host to MIT. The ‘smoke’ around the green line indicates variance in the ping times. If more smoke is above the line, there was more occurances of longer ping times, if more smoke is underneath the colored line, there were ping times that came quicker than the average At around 1040am, we see ping times from 9ms to 16ms, the average being where the green line is drawn. Green indicating there was 0% packet loss.

An example of the multigraph is:

What we see here is fairly stable latency to MIT and Stanford, but fluctuating latency to Univ of Texas. Latencies range from about 10ms to MIT (5 miles), 75-140ms to Univ of Texas (2000 miles), and 115ms to Stanford (3100 miles). Packet loss is minimal on all traces, as we see “0.0% av ls” on all three data points.

Hopefully this has been helpful for those who find this post, and a reminder for me on how I got things working for further installations (and re-installations) of Smokeping.

[1] AT&T Network Latency: http://ipnetwork.bgtmo.ip.att.net/pws/network_delay.html