Stack it up: KVM, VLANs, Network Bridges, Linux/OpenBSD

I’ve had some free time and a desire to break stuff on my network at home. I wanted to fix my home network’s topology to more correctly split up my wired (DHCP), wireless (DHCP) and server (statically-configured) subnets. At a high level, I had to create a server subnet, create vlan’s on my layer-3 switch for each of those pervious subnets, then I had to move the network interfaces on my VM host around to only connect to the networks I wanted it to (wired and server).

First, I moved the secondary interface of my VM host at home from the wifi network to the new server network. The server network would have its own subnet ( and its own VLAN, Vlan 12. (we all have Layer3 managed-switches at home, right?).

The fw/router on my network runs OpenBSD.  Interface ’em3′ will be providing connectivity to  vlan 12.

$ cat /etc/hostname.em3 up description "Server Iface"  $ cat /etc/hostname.vlan12 inet vlan 12 vlandev em3 description "Server Network vlan12" 

Depending on if I want to throw more VLAN’s on em3, I can just create more hostname.vlan{number} files with appropriate network configuration.

The switch managing my network’s traffic is an HP 2910al-24G (J9145). The relevant configuration to handle tagged packets on vlan-12 on two ports (4, 12) is:

vlan 12 name "Servers-10-10-2-0-24" tagged 4,12 ip address exit 

I’ve also added a management IP on this server subnet VLAN as well. On my VM host, this took me the most time to get right. The hardware network interface for the server network is eth1. I wanted to create a bridge on this interface so that multiple VM’s could use this interface to bind to. This bridged interface on vlan-12 is br-vlan12. This host also has an IP address on the network itself so that I can access the VM host itself over the server subnet.

auto eth1 iface eth1 inet manual  auto br-vlan12   iface br-vlan12 inet static   address   network   netmask   broadcast   bridge_ports eth1.12   bridge_stp off 

One of the pieces that had me fumbling for so long is getting the bridge_ports and vlan_raw_interfaces specification on the bridge. It turns out when the former is specified in a certain way, the latter is not needed.

From interfaces(5):

VLAN AND BRIDGE INTERFACES To ease the configuration of VLAN interfaces, interfaces having . (full stop character) in the name are configured as 802.1q tagged virtual LAN interface. For exam‐ ple, interface eth0.1 is a virtual interface having eth0 as physical link, with VLAN ID 1.  For compatibility with bridge-utils package, if bridge_ports option is specified, VLAN interface configuration is not performed. 

I am still having some fun issues with routing on my network, where I can ping from my wifi network to the wired-network interface on my LAN, but not wifi -> server. I think this has to do with reverse path forwarding (RPF) checking on the server, given its default route is over the ‘wired’ network and not the server network interface. An invaluable tool to debugging these types of issues has been the sysctl setting below, logging martians. It logs instances where packets come into an interface which it does not expect, and therefore by-default rejects.

net.ipv4.conf.all.log_martians = 1 

The fun and breakage continues.

Helpful links found during my foray into this topic:

Unattended Ubuntu installs, part 2

In my initial post about unattended Ubuntu installs, I made the less-automated choice of hacking at the Ubuntu installation ISO and baking my preseed configuration right into the ISO. This proved to be incredibly inefficient and prevented a lot of the customization and quick-spin-up potential of what I interested in. In other words, if I wanted to spin up five identical VMs differing only by their hostname, was I really expected to bake five custom ISO’s whose preseed file only differed by their specification of the hostname?

Solution: With a bit of Internet poking, I found that you can specify the URL of a preseed file, accessible via HTTP, for your VM to read during OS installation as a kernel boot parameter. Given all this, there really was no reason to bake my own ISO in the first place. I had to test using virt-install specifying all these parameters on the command line, including using a straight Ubuntu install ISO. Results? Success!

For those curious of the command-line I used:

sudo /usr/bin/virt-install    --connect qemu:///system    --disk vol=<disk pool>/<disk volume>,cache=none    --extra-args "locale=en_US.UTF-8 console-keymaps-at/keymap=us console-setup/ask_detect=false console-setup/layoutcode=us keyboard-configuration/layout=USA keyboard-configuration/variant=US netcfg/get_hostname=<VM hostname> netcfg/get_domainname=<VM domain name> console=tty0 console=ttyS0,115200n8 preseed/url=<URL to preseed file>"    --location /mnt/raid1/dump/ubuntu-14.04.1-server-amd64.iso   --network bridge=<bridge if>   --name <VM name according to libvirt>    --os-type linux   --ram 512    --vcpus 1    --virt-type kvm  

Preseed file: This file can live on any accessible-from-your-VM http server. During the install process, it is retrieved via wget as part of the install procedure. But how do you specify the URL for the preseed file?

The only one modification I did have to make to my preseed file had to do with selecting a mirror. I was constantly prompted to select a mirror hostname. After another couple Google searches, I was left with what seems to work, by default picking a US-based HTTP mirror for Ubuntu packages:

d-i mirror/http/countries select US d-i mirror/http/hostname string d-i mirror/http/directory string /debian d-i mirror/http/mirror select d-i mirror/protocol select http d-i mirror/country string US d-i mirror/http/proxy string 


Look ma’, no hands with Ubuntu installs.

In my day job, it’s all about automation. Automate what is repeatable, and move on to more interesting and not-yet-automated tasks. For a while, I’ve run a KVM/libvirt setup at home, running various iterations and distributions of Linux, OpenBSD and FreeBSD for various pet projects. Going through each distribution’s install procedure was getting old, requiring me to input the same parameters, set up the same users and passwords, over and over again. Given I use Ubuntu mostly as a VM guest, I dug into their preseed infrastructure, to be able to automate the installation and get me past the drudgery of adding another VM. Below are the steps and a bit of sample configuration that got me through the process.

I did find some examples of automating this all the way from virt-install (libvirt’s way of adding a VM instance to your cluster), but that is for another time.

[Update 2014-09-13: Even more unattended. Part 2]

Grab a Ubuntu Server ISO from their web site. Mount the ISO locally and rsync its contents to a new directory for your own customization.

mount -o loop /path/to/iso /some/mountpoint rsync -av /cdrom/ /opt/cd-image 

Now we’re left with the customization of the install. I wanted the installation to be completely hands-free. I shouldn’t have to enter in any partition information, user names, or network information. Right now my parameters are, for the most part, configured in the preseed file. My eventual goal is the factor those out into my own personal install script so that the command line arguments from my script are passed as kernel options to the install and are read at run-time as opposed to at CD-creation time. Doing it that way alleviates the need to re-create a new ISO with hard-coded values for the host name, domain name, and various network information, among others, each time you want to build a new VM in the preseed file.

/opt/cd-image/isolinux/txt.cfg (additions):

LABEL forman-preseed menu label ^Forman Preseed kernel /install/vmlinuz append preseed/file=/cdrom/preseed/ubuntu-server-custom.seed vga=788 initrd=/install/initrd.gz locale=en_US.UTF-8 console-keymaps-at/keymap=us console-setup/ask_detect=false console-setup/layoutcode=us keyboard-configuration/layout=USA keyboard-configuration/variant=USA -- 


d-i debian-installer/locale string en_US d-i netcfg/choose_interface select eth0 d-i netcfg/get_hostname string preseedhost-1 d-i netcfg/get_domain string foobar.mylan d-i netcfg/wireless_wep string  d-i time/zone string US/Eastern d-i clock-setup/ntp boolean true  d-i partman/choose_partition select finish d-i partman/confirm boolean true d-i partman/confirm_nooverwrite boolean true d-i partman/default_filesystem string ext4 d-i partman-auto/init_automatically_partition select biggest_free d-i partman-auto/choose_recipe select atomic d-i partman-auto/method string regular d-i partman-auto/select_disk string /dev/vda d-i partman-md/confirm boolean true d-i partman-partitioning/confirm_write_new_label boolean true  d-i passwd/root-password-crypted password <echo "foo" | mkpasswd -m md5 -> d-i passwd/user-fullname string FirstName LastName d-i passwd/username string myfirstuser d-i passwd/user-password-crypted password <echo "foo" | mkpasswd -m md5 -> d-i user-setup/allow-password-weak boolean true d-i user-setup/encrypt-home boolean false  d-i mirror/http/proxy string d-i pkgsel/include string openssh-server irssi d-i pkgsel/upgrade select full-upgrade d-i pkgsel/update-policy select none  tasksel tasksel/first multiselect basic-ubuntu-server  d-i clock-setup/utc boolean true  d-i grub-installer/only_debian boolean true d-i grub-installer/timeout string 2 d-i finish-install/keep-consoles boolean true d-i finish-install/reboot_in_progress note 

This creates a VM with the various properties (highlights for brevity):

  • hostname = preseed-host1
  • domain name = foobar.mylan
  • network = DHCP via eth0
  • partitioning scheme = one big / partition, with space leftover for SWAP
  • one user with password set, along with root’s password.
  • install openssh-server and irssi to confirm package installation works.

Create the CD image:

IMAGE=custom.iso BUILD=/opt/cd-image/  mkisofs -r -V "Custom Ubuntu Install CD"    -cache-inodes    -J -l -b isolinux/isolinux.bin    -c isolinux/    -no-emul-boot    -boot-load-size 4    -boot-info-table    -o $IMAGE    $BUILD 

Voila. Boot that as your install CD and behold the magic! Booting from this ISO inside your VM instance should leave you with a fully functionining instance.

The only real hiccups I hit along the way, given the multitude of documentation out there on the Internet, was getting past the keyboard selection prompts. Specifying the keyboard model, layout, and ‘ask_boolean false’ values for that set of questions inside the preseed file had no affect, I was still prompted. Those configuration values seemed to be REQUIRED to be set in the isolinux config, which in my case I stuck in txt.cfg.

Sources (of inspiration):

i3wm, i3bar, and rhythmbox

I was interested in customizing my i3wm setup a bit more, and wanted to display the current song playing in Rhythmbox while running the i3wm window manager.  It turned out to be just a few lines of configuration to my i3bar config.

First, I grabbed a copy of the Python wrapper around i3bar, This wrapper merely takes the output of a command, wraps it in compliant JSON, and returns in a way that i3bar uses it generate its output. The wrapper allows the user to add on an arbitrary number of additions to their status bar, although you don’t want to overwhelm what is supposed to be just a status bar.

My own code I added to

import subprocess  def get_rhythmbox_status():     """ Get the current song playing and elapsed time from rhythmbox if it is running. """     cmd = "rhythmbox-client --no-start --print-playing-format '%aa - %tt (%te)'"    try:        rb_output = subprocess.check_output(cmd, shell=True)        rb_output = rb_output.strip()    except subprocess.CalledProcessError, cpe:        rb_output = "Rhythmbox Client: Error"     if "(Unknown)" in rb_output:        rb_output = "Rhythmbox: Not Playing"     return rb_output 

This function executes rhythmbox-client. Command line flags include purposely not starting Rhythmbox if it is not started, and displaying a custom format of output related to the current song being played. This output is the artist name, track title, and elapsed time of the current track. If Rhythmbox is not started, no output is present in the i3bar. If Rhythmbox is started but no song is playing, “Rhythmbox: Not Playing” is displayed.

in the ‘main function’, I modified the line which inserts an entry in the JSON object:

j.insert(0, {'full_text' : get_rhythmbox_status(), 'name' : 'rhythmbox'}) 

Inside my i3wm config’s bar specification, my status_command line looks like the following:

status_command i3status --config ~/.i3/i3status.conf | ~/.i3/ 

If your i3bar updates on an interval, you will see the current elapsed time of the song update as it plays.

FreeBSD on the desk, another try

After several years of mindlessly running Ubuntu on the desktop, I am attempting to dive (back) into running FreeBSD on the desktop. Considering that the majority of applications I use on the desktop are a browser (Firefox/Chrome), an ssh terminal, and Rhythmbox, how hard could this be?

Some of the hurdles

Given I still wanted to keep Ubuntu around and not redefine my default setup, I kept Grub2 as my bootloader on the MBR. I still needed a way to boot into FreeBSD at-will. I had installed FreeBSD on hd0a. Grub2 from Ubuntu makes finding the FreeBSD boot files incredibly easy:

>search -f /boot/kernel hd0, msdos2 > set root=(hd0, msdos2) > chainloader +1 > boot 

Considering it has been a while since I ran FreeBSD for anything serious, I had always debated between ports and packages. In my distant memory, packages were not built for every piece of software I wanted, and building ports has the downside of long compile times, and potentially hairy dependency issues. With FreeBSD 10.0-RELEASE, ‘pkg’ has become the default package manager, and thus far, I have had no issues finding packages for any software package I’ve wanted. Upon install, update the package repository:

# pkg update 

and use the various ‘pkg search’ and ‘pkg install’ variants to search for, and install the various applications.

I’ve always been curious in the various tiling window managers. i3wm seems to have the most-sane configuration structure among the various other tiling window managers (xmonad, awesomewm, etc). My i3wm configs up on Github.

I am still working on making the tiling-window manager thought process more second-nature. One instance I’m still attempting to wrap my head around, is when I fire up a full-screen window from Chrome, which ends up ‘under’ my main Chrome window. This makes the refrain ‘where the heck did my window go’ quite common. Alongside the fact that there is a lot of font configuration and xorg.conf hacking required to make the desktop what I consider ‘pretty.’

I find myself booting back into Ubuntu more often than not, given I’m more comfortable with the Unity window manager workflow. But I do boot into FreeBSD when time permits to try and hack on making it an actual usable desktop OS. The journey continues.

Wireless, now with more 802.11’s…

With nothing else to do around here tonight while the whole state is shut down thanks to a blizzard, I should catch up on some blog posts.

On my list of home network upgrades for the past several months was the wireless. As my wife and I add to our collection of smart phones, laptops, tablets, and wireless streaming devices (I am looking at you EOL Logitech Revue with Google TV) the amount of latency and available bandwidth started to show signs of strain. I had been running the wireless for several years on an Asus WL-500G Premium v2 router/wap, which only ran 802.11b/g over 2.4Ghz. It was time for an upgrade.

Welcome our new Asus RT-N66U 802.11b/g/n router/wap that handles dual channel 2.4Ghz and 5Ghz wifi.

I did some very unscientific comparisons before and after I performed the hardware upgrade. I pushed and pulled a ~763MB Ubuntu ISO across the wireless, through a 10/100Mb switch, via rsync over SSH from a server on the LAN. The following table shows rsync’s average speed and transfer duration from the point of view of a 15″ Macbook Pro connected via the wifi.

Old Wifi New Wifi 2.4Ghz New Wifi 5Ghz
Upload  1.87MB/s (6:46)  7.69MB/s (1:39)  5.57MB/s (2:17)
Download  2.61MB/s (4:51)  10.81MB/s (1:10)  10.39MB/s (1:13)

Needless to say, I am keeping the new router.

California 2012, thricely.

Condensed version of trip #3 to California.

San Diego

  • Sushi Ota: Sake and sushi with Mozilla folks. For the quality of the sushi (incredible), the price (reasonable) blew me away.
  • Tajima: Ramen! The spicy miso ramen here lives up to its name, be prepared.
  • Fish Market
  • Cucina Urbana: Serious Italian, and a wine list to match. Funky interior too.
  • Davanti Enoteca: Good tripe dish

San Francisco

  • State Bird Provisions: Dim sum delivery, California style.
  • Black Point Cafe: Coffee refuel near the Golden Gate Bridge. Killer latte. Gary Danko
  • Spruce: The duck. Yes, get the duck.
  • Philz Coffee: No justification needed. Had to restock the East Coast supply.
  • Acqurello
  • Izakaya Roku: More ramen! Sake!
  • Humphry Slocombe: Blue Bottle Vietnamese Coffee Ice Cream. Good lord. I can now die a happy man.
  • Incanto: Charcuterie zen master.

Napa/St Helena

  • White Rock Winery: We happened upon this one as they were dumping out the lower quality wine. So sad.
  • Oakville Grocery Co: Mid-Napa refuel. Bread, cheese, meats.
  • Saddleback Cellars
  • Gott’s Roadside: The juxtaposition of this in Napa is pretty jarring. Their milkshakes are a nice divider between all the wine.
  • Stag’s Leap Wine Cellars: This is where I cemented the fact that I like the smaller wineries, rather than the commercial behemoths.
  • Cliff Lede Winery
  • Morimoto Napa

Central California Coast

  • La Bicyclette (Carmel): Pizza, charcuterie, cheese. Take out. To be returned.
  • Sierra Mar at Post Ranch Inn (Big Sur): One of the most spectacular sunset views on the Pacific Coast. Food is damn tasty too.
  • Big Sur Bakery: Brunch doesn’t start until 1030am, but the fruit strudel and latte’s are killer.
  • Dover Canyon Winery: Our first foray into Paso Robles wines. Unexpectedly awesome. Along with their 185 lb. St Bernard named Thunder.
  • Turley Winery
  • Whalebone: Free chili with every tasting. Clutch with all the damn rain.
  • Adelaide Winery
  • Olavino: Olive oil and salt, one made with ghost chili. Holy crap. and holy good.
  • L’Adventure Winery

Nagios and Git hooks, a redux

A while back I blogged about how I hooked up Nagios and Git to run the Nagios preflight checks before restarting with a new checkin’s worth of configs. But the more I looked at how it all fit together, the more I knew it could be improved. A sed hack, expecting a certain pattern in the nagios.cfg? Bad bad bad. Most of the improvement revolves around Nagios’s ability to reference relative paths for its config files. Given the path of the ‘main’ nagios.cfg file, you can then reference directories that contain your services, hosts, and other custom commands, in relation to that main file. With this functionality I significantly improved the Git->Nagios pipeline.

First, the pre-receive hook

#!/bin/bash  umask 022  while read OLD_SHA1 NEW_SHA1 REFNAME;  do  export GIT_WORK_TREE=/tmp/nagiostest-$NEW_SHA1  mkdir -p $GIT_WORK_TREE /usr/bin/git checkout -f $NEW_SHA1 sudo /usr/sbin/nagios3 -v $GIT_WORK_TREE/nagios.cfg if [ "$?" -ne "0" ]; then  echo "Nagios Preflight Failed"  echo "See the above error, fix your config, and re-push to attempt to update Nagios."  exit 1  else  echo "Nagios Preflight Passed"  echo "Clearing temporary work directory."  rm -rf $GIT_WORK_TREE exit 0 fi done 

Using the GIT_WORK_TREE environment variable, which specifies Git’s working directory, I check out the new set of potential configs to a temporary directory. This provides a temporary ‘waiting room’ for the proposed configuration to be tested, before before being put into production. Imagine never (intentionally) breaking Nagios again because of a broken host or service specification. The main thing remember is that all references in the nagios.cfg to other config files (hosts, commands, etc) must be relative paths. I.E., I have lines that look like “cfg_dir=configs” in the nagios.cfg. Note the lack of absolute paths. We now run the Nagios pre-flight check (nagios -v) on the nagios.cfg in the Git work tree. Depending upon the exit value of ‘nagios -v’, 0 for success and 1 for failure, we either proceed or die immediately. If success, clean up our temporary run directory.

Now the post-receive hook:

#!/bin/sh  echo "Updating repo /etc/nagios3" sudo /usr/bin/update-gitrepo /etc/nagios3 

The post-receive hook merely runs a script, noted below, on the Nagios configuration directory.


#!/bin/sh umask 022  REPO_DIR=$1 cd ${REPO_DIR}  /usr/bin/git pull origin master 

Given the Git checkout’s directory, we fetch the most recent push to the repository.

For the final step we have to fix some permissions (given that my setup runs the repository through Gitolite as the git user). This hook is located in the actual checkout itself, /etc/nagios3, in the post-merge hook.

#!/bin/sh  sudo chown -R nagios:admin /etc/nagios3 sudo /etc/init.d/nagios3 restart 

A full commit and restart looks like this:

jforman@merlot:/mnt/raid1/personal/git/monitor/nagios/configs$ git push Counting objects: 7, done. Delta compression using up to 4 threads. Compressing objects: 100% (4/4), done. Writing objects: 100% (4/4), 414 bytes, done. Total 4 (delta 3), reused 0 (delta 0) remote: Previous HEAD position was c80fa03... turn off test notifications with notifications_enabled 0 remote: HEAD is now at f088dbc... Example: Add boilerplate header that file is managed by Git. remote: remote: Nagios Core 3.2.3 remote: Copyright (c) 2009-2010 Nagios Core Development Team and Community Contributors remote: Copyright (c) 1999-2009 Ethan Galstad remote: Last Modified: 10-03-2010 remote: License: GPL remote: remote: Website: remote: Reading configuration data... remote: Read main config file okay... remote: Processing object config file '/tmp/nagiostest-f088dbcebf194edbce78068b6004cbbfca703432/commands.cfg'... remote: Processing object config directory '/etc/nagios-plugins/config'... remote: Processing object config file '/etc/nagios-plugins/config/ftp.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/mail.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/snmp_int.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/nt.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/http.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/real.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/check_nrpe.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/snmp_storage.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/disk.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/mysql.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/snmp_load.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/fping.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/dhcp.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/ssh.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/rpc-nfs.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/mailq.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/breeze.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/dummy.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/netware.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/hppjd.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/load.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/mrtg.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/apt.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/snmp_cpfw.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/snmp.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/snmp_process.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/snmp_env.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/news.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/ntp.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/telnet.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/users.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/snmp_mem.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/procs.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/ifstatus.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/games.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/disk-smb.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/tcp_udp.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/snmp_win.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/ping.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/pgsql.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/ldap.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/flexlm.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/dns.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/radius.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/snmp_vrrp.cfg'... remote: Processing object config directory '/tmp/nagiostest-f088dbcebf194edbce78068b6004cbbfca703432/configs'... remote: Processing object config file '/tmp/nagiostest-f088dbcebf194edbce78068b6004cbbfca703432/configs/timeperiods.cfg'... remote: Processing object config file '/tmp/nagiostest-f088dbcebf194edbce78068b6004cbbfca703432/configs/services.cfg'... remote: Processing object config file '/tmp/nagiostest-f088dbcebf194edbce78068b6004cbbfca703432/configs/commands.cfg'... remote: Processing object config file '/tmp/nagiostest-f088dbcebf194edbce78068b6004cbbfca703432/configs/hosts.cfg'... remote: Processing object config file '/tmp/nagiostest-f088dbcebf194edbce78068b6004cbbfca703432/configs/abstracts.cfg'... remote: Processing object config file '/tmp/nagiostest-f088dbcebf194edbce78068b6004cbbfca703432/configs/contacts.cfg'... remote: Read object config files okay... remote: remote: Running pre-flight check on configuration data... remote: remote: Checking services... remote: Checked 95 services. remote: Checking hosts... remote: Checked 10 hosts. remote: Checking host groups... remote: Checked 7 host groups. remote: Checking service groups... remote: Checked 0 service groups. remote: Checking contacts... remote: Checked 3 contacts. remote: Checking contact groups... remote: Checked 2 contact groups. remote: Checking service escalations... remote: Checked 0 service escalations. remote: Checking service dependencies... remote: Checked 56 service dependencies. remote: Checking host escalations... remote: Checked 0 host escalations. remote: Checking host dependencies... remote: Checked 0 host dependencies. remote: Checking commands... remote: Checked 181 commands. remote: Checking time periods... remote: Checked 4 time periods. remote: Checking for circular paths between hosts... remote: Checking for circular host and service dependencies... remote: Checking global event handlers... remote: Checking obsessive compulsive processor commands... remote: Checking misc settings... remote: remote: Total Warnings: 0 remote: Total Errors: 0 remote: remote: Things look okay - No serious problems were detected during the pre-flight check remote: Nagios Preflight Passed remote: Clearing temporary work directory. remote: Updating repo /etc/nagios3 remote: From monitor:nagios remote: * branch master -> FETCH_HEAD remote: Updating c80fa03..f088dbc remote: Fast-forward remote: configs/commands.cfg | 2 ++ remote: 1 file changed, 2 insertions(+) remote: * Restarting nagios3 monitoring daemon nagios3 remote: Waiting for nagios3 daemon to die.. remote: ...done. To git@monitor:nagios.git c80fa03..f088dbc master -> master 

Note that I do keep the Nagios package bundled commands in the /etc/nagios-plugins directory and have purposely not put those in the Git tree. This allows for updated Nagios packages from Ubuntu to update those commands accordingly without interfering with the Git repo.


Remind Me: Damage done in San Francisco in six days

I am lucky enough to have a sister living out in San Francisco, and to be able to work out of our offices there. Below is a hit list of the places I ate at and visited in the span of six days. My stomach has finally recovered.

Wineries (Sonoma County):

Both of us are lucky enough to have been through Napa several times, so we decided to venture into Sonoma County. The last item in that list is an unsuspecting general store off of Dry Creek Road in Heldsburg which has an incredible sandwich list. This area turns Napa on its head, with a much more family-run low-key atmosphere. There is none of the pretense of visiting a large production winery such as Mondavi or the herds of people who visit Duckhorn.

To Eat:

I cannot recommend every one of these places enough. Izakaya Sozai serves killer ramen. Yank Sing serves dim sum on weekend mornings that melt in your mouth. Shuck your own oysters at Hog Island (we learned in about 30 seconds) at the farm while sitting on benches along Tomales Bay. Mission Chinese blasts gangster rap while you gorge yourself on craveable Chinese food.

Go, eat, recover later.

The Internet is slow. Is the Internet down?

We have all heard the same questions at one point in our careers, “Is the Internet down?” or “Getting to X site is slow.” You scramble to a browser to see if Google, ESPN or the NY Times websites are up. Then you fire up traceroute. In some cases, the pages might load slowly, in other cases not at all. These two situations are often downstream fallout of two connectivity issues: latency and packet loss. Latency is the time it takes for a packet to get from source to destination. The speed of light says the latency for one packet to get across the USA from New York to San Francisco is normally between 70-90ms [1]. Packet loss occurs when packets do not make it from their source to destination, being lost along the way. Many factors can contribute to packet loss, including overloaded routers and switches, service interruptions, and human error.

When diagnosing network issues between source and destination, it is helpful to have data to backup your suspicions of slow and inconsistent network performance. Insert Smokeping.

As part of a network and system monitoring arsenal, you might have Nagios configured for host and service monitoring, and Munin for graphing system metrics. But for monitoring network performance, I feel Smokeping fills that gap. Below are some notes I took getting Smokeping installed and running on a Ubuntu Linux VM at home.

I installed Smokeping from source, since the version in the Ubuntu repository (2.3.6 for Ubuntu Oneiric) is quite old compared to the latest release, 2.6.8, at the time of this post. After installing the various dependencies from the Ubuntu repo, I was able to build and install Smokeping under /opt/smokeping. One thing I do appreciate about Smokeping is that you can run it as any arbitrary user. No root needed!

First we need to configure Smokeping and verify it starts up.

Part of my Smokeping config:

imgcache = /opt/smokeping/htdocs/cache   imgurl   = http://yourserver/smokeping/cache  + random   menu = random   title = Random Hosts  ++ utexas   host =  ++ stanford   host =  ++ mit   host =  ++ multihost   title = edu comparison   host = /random/utexas /random/stanford /random/mit   menu = EDU Host Comparison 

imgcache must be the absolute path on your webserver where Smokeping’s cgi process writes out png files. imgurl is the absolute URL where your httpd presents the imgcache directory.

What follows is a sample stanza under the ‘charts’ category in the config. It contains three discrete Smokeping graphs to webservers found at the MIT’s Media Lab, University of Texas, and Stanford University. I picked these three hosts because they represent a variety of near, far, and trans-continental servers from my home in the Northeastern US. The last entry, multihost, creates one single graph with the three data points combined. The ‘host’ parameter in this case contains three path-like references to the graphs we want consolidated into one graph.

To test that Smokeping starts up, execute the following:

jforman@testserver1 /opt/smokeping % ./bin/smokeping --config /opt/smokeping/etc/config --nodaemon   Smokeping version 2.006008 successfully launched.   Not entering multiprocess mode for just a single probe.   FPing: probing 3 targets with step 300 s and offset 161 s. 

When you are ready to take the training wheels off, remove the ‘–nodaemon’ argument, and put this command in your distribution’s rc.local file to be started at boot time.

To actually view the generated data in graphs, you will need CGI support configured in your httpd of choice. For the most part, I run Apache.

Snippets of required Apache configuration:

LoadModule cgi_module /usr/lib/apache2/modules/   AddHandler cgi-script .cgi  Alias /smokeping "/opt/smokeping/htdocs"   <Directory "/opt/smokeping/htdocs">   Options Indexes MultiViews   AllowOverride None   Order deny,allow   Allow from all   Options ExecCGI   DirectoryIndex smokeping.cgi   </Directory> 

I am not presenting my Smokeping install as a virtual host, so I have left that part out. Also take note that the httpd’s user needs to have permissions on the imgcache directory in your Smokeping config file. In my case, /opt/smokeping/htdocs/cache is 775 with www-data as the group.

Hopefully this has been helpful for those who find this post, and a reminder for me on how I got things working for further installations (and re-installations) of Smokeping.

[1] AT&T Network Latency: