Family Tech Support: Vacation Edition

This was an epic visit home, tech-wise. Just so I don’t forget, and can hold it over my folks’ head for a while:

  • Upgraded two five-year-old Linksys E2000 AP’s to Netgear r6250’s. Those old ones were just not reaching the entire length of the house anymore.
  • Upgraded the firewall/router from OpenBSD 5.5-stable to OpenBSD 5.6-stable. It just so happens I’m home every six months to stay relatively close to the most-recent errata.
  • Converted my father’s Gmail account over from one-factor to two-factor authentication thanks to some nasty spyware/adware and potential identity-theft issues he’s had recently. I wasn’t willing to do this conversion remotely given the horror of Application Specific Passwords and how many devices I would have to do it on (desktops, laptops, one iPhone, and one iPad)
  • Reinstalled one Late 2009 21.5″ iMac via Internet Recovery to OSX 10.10 due to aforementioned nasty adware infestation.
  • Upgraded that same iMac from 4GB RAM to 16 GB RAM.

All I can say is that it’s nice having all Mac’s in the house now, after finally kicking out the last Windows-based PC on my last visit.

Third time’s a charm? Gitolite, Git, Nagios, and a bunch of hooks

I was hoping with my past posts on this topic, I would have enough examples to just copy-and-paste along to configure my Gitolite+Nagios monitoring setup. Not so true. It looked like there were semi-colon’s missing in my past examples. After looking at the huge number of changes in Gitolite, I had to re-do everything. Not to mention I always wanted a better way to manage the hooks as opposed to editing them directly on the host. In short, my goal is still simple: be able to manage and verify Nagios configuration remotely via Git. Below is how I did it. For the third time.

First, install Gitolite. I run Gitolite under the ‘git’ user on my Ubuntu-based VM, from now on called monitor1. I clone the Gitolite source under /home/git/gitolite.

In /home/git/.gitolite.rc, in the %RC block, uncomment:

LOCAL_CODE => "$rc{GL_ADMIN_BASE}/local", 

This option tells Gitolite we have some local configuration in our gitolite-admin repository under the ‘local’ directory. More on this later.

In the ENABLE list, uncomment:

'repo-specific-hooks', 

This option tells Gitolite we want to be able to to use repo-specific hooks, as opposed to having one set of hooks for all repositories.

Since several of our yet-to-be-defined hooks need elevated permissions, I have configured a sudoers file to allow so.

%nagios-admin ALL=(ALL) NOPASSWD: /usr/sbin/nagios3 -v /tmp/nagiostest-*/nagios.cfg %nagios-admin ALL=(ALL) NOPASSWD: /usr/sbin/service nagios3 restart 

Our ‘nagios’ user is added to the nagios-admin group, along with the git user. This allows us via Gitolite’s hooks to test, update, and restart the Nagios installation.

This concludes all the work on monitor1 as it relates to Gitolite.

On your local workstation, clone the gitolite-admin repo. I’ve chosen to name the repo containing my Nagios configuration, ‘nagios’. At this point, it is probably safe to copy a known-working copy of your Nagios configuration files into the nagios repository itself. The steps following here, if done in one fell swoop, could completely blow away your /etc/nagios3 directory on monitor1 if you are not careful.

One modification necessary for the nagios.cfg itself is to modify the references to the path of the the configuration files. By default, the nagios.cfg lists an absolute path to the files, e.x: /etc/nagios3/conf.d/. In our case, we will be checking out the configuration files to a temporary directory while we run our pre-flight checks and need to use a relative path instead to make this possible.

Therefore in your nagios.cfg file, perform the following changes:

cfg_file=commands.cfg cfg_dir=conf.d 

Now that I look at it, I’m not quite sure why these are specified separately, as your commands.cfg file could live under conf.d. But I’ll leave that for readers who have their own structure preferences. The key here is that relative paths must be used, NOT absolute ones.

Next we move onto the gitolite-admin configuration:

repo nagios   RW+ = jforman_rsa   option hook.pre-receive = nagios-pre-receive   option hook.post-receive = nagios-post-receive   option hook.post-update = nagios-post-update 

This tells Gitolite the name of my nagios config repository, who is ACL’d to read and write to it, and which hooks I wish to override with my custom hooks. Note that in Gitolite, you can only override these three hooks, pre and post-receive, and post-update. Other hooks such a post-merge and merge are special to Gitolite and you will be returned an error if you attempt to override them. Note that each hook, nagios-pre-receive, and so on, corresponds to a file name that will live under my gitolite-admin repository under the ‘local’ directory.

Now we come to the point of defining our hooks. Under your gitolite-admin directory, create the directory structure ‘hooks/repo-specific’ under the directory we defined in the above LOCAL_CODE definition. In our case, that corresponds to ‘local’.

In other words. in our local checkout of the gitolite-admin repository:

mkdir -p ${gitolite_admin_path}/local/hooks/repo-specific 

Under this repo-specific directory, using whatever language you prefer (Python, Shell, Perl, etc), create the files for the repository’s hooks.

gitolite-admin$ tree local/ local/   └── hooks     └── repo-specific       ├── nagios-post-receive       ├── nagios-post-update       └── nagios-pre-receive 

nagios-pre-receive:

#!/bin/bash  umask 022  while read OLD_SHA1 NEW_SHA1 REFNAME;  do  export GIT_WORK_TREE=/tmp/nagiostest-$NEW_SHA1  mkdir -p $GIT_WORK_TREE /usr/bin/git checkout -f $NEW_SHA1 sudo /usr/sbin/nagios3 -v $GIT_WORK_TREE/nagios.cfg if [ "$?" -ne "0" ]; then   echo "Nagios Preflight Failed"    echo "See the above error, fix your config, and re-push to attempt to update Nagios."    exit 1  else    echo "Nagios Preflight Passed"    echo "Clearing temporary work directory."    rm -rf $GIT_WORK_TREE exit 0 fi done 

nagios-pre-receive: Using the most recent commit, checkout this body of work into a temp directory and run the Nagios pre-flight checks over it. If those checks pass, exit 0 (without error). If those pre-flight checks fail, error out. This latter case will stop the Git push completely. Your running Nagios configs in /etc/nagios3 are untouched.

nagios-post-recieve

#!/bin/bash  echo "Updating repo /etc/nagios3" /usr/bin/update-gitrepo /etc/nagios3 

nagios-post-receive: This runs a companion script to update the cloned git repository at /etc/nagios3. Note that this is only run if the post-receive succeeds, and executes after the merge step of the git push has succeeded, which means our Gitolite nagios repository has now merged the commits we are attempting to push.

nagios-post-update

#!/bin/bash  sudo chown -R root:nagios-admin /etc/nagios3 sudo /usr/sbin/service nagios3 restart 

nagios-post-update: The post-update step runs after the post-receive, ensuring our permissions are correct and then restarts Nagios.

At this point, the custom hooks and gitolite.conf should be committed and pushed to the remote Gitolite gitolite-admin repository. No more storing hooks in the bare repository itself! The hooks themselves are version controlled. This was what bugged me the most about my prior solutions. I always hated not having any history on how I fixed (broke) the scripts in the past.

update-gitrepo

#!/bin/bash umask 022  REPO_DIR=$1 cd ${REPO_DIR} unset GIT_DIR /usr/bin/git pull origin master 

update-gitrepo: Lives on the monitor1 under /usr/bin, and merely executes a ‘git pull’ under the passed directory.

This last bit of configuration I’m not too happy with, given it feels so manual and hacky. It’s how you get /etc/nagios3 to be the Git repository checkout itself. On monitor1, I normally do this initial work in /tmp, as the ‘git’ user.

cd /tmp git clone /home/git/repositories/nagios.git/ (as root) mv /etc/nagios3 /etc/nagios3.notgit (as root) mv /tmp/nagios /etc/nagios3 (as root) chown -R root:nagios-admin /etc/nagios3 /usr/bin/update-gitrepo /etc/nagios3 

This performs a checkout of the Nagios repository itself (note that we’re skirting Gitolite’s ACL control and accessing the repository directly on the file system). If you wish to further control who can check out the nagios directory, create an ssh-key for your ‘git’ user locally, add to the gitolite-admin repository ACL, and perform your checkout over SSH as opposed to using the absolute-path of the directory.

Wow. This post turned out to be much longer than I expected. If you’ve made it this far, you should have the groundwork laid to do remote clones of your Nagios configuration files, and have the ability to run the pre-flight check verify their correctness before ever getting near your production Nagios directory. Good luck.

Stack it up: KVM, VLANs, Network Bridges, Linux/OpenBSD

I’ve had some free time and a desire to break stuff on my network at home. I wanted to fix my home network’s topology to more correctly split up my wired (DHCP), wireless (DHCP) and server (statically-configured) subnets. At a high level, I had to create a server subnet, create vlan’s on my layer-3 switch for each of those pervious subnets, then I had to move the network interfaces on my VM host around to only connect to the networks I wanted it to (wired and server).

First, I moved the secondary interface of my VM host at home from the wifi network to the new server network. The server network would have its own subnet (10.10.2.0/24) and its own VLAN, Vlan 12. (we all have Layer3 managed-switches at home, right?).

The fw/router on my network runs OpenBSD.  Interface ’em3′ will be providing connectivity to  vlan 12.

$ cat /etc/hostname.em3 up description "Server Iface"  $ cat /etc/hostname.vlan12 inet 10.10.2.1 255.255.255.0 10.10.2.255 vlan 12 vlandev em3 description "Server Network vlan12" 

Depending on if I want to throw more VLAN’s on em3, I can just create more hostname.vlan{number} files with appropriate network configuration.

The switch managing my network’s traffic is an HP 2910al-24G (J9145). The relevant configuration to handle tagged packets on vlan-12 on two ports (4, 12) is:

vlan 12 name "Servers-10-10-2-0-24" tagged 4,12 ip address 10.10.2.15 255.255.255.0 exit 

I’ve also added a management IP on this server subnet VLAN as well. On my VM host, this took me the most time to get right. The hardware network interface for the server network is eth1. I wanted to create a bridge on this interface so that multiple VM’s could use this interface to bind to. This bridged interface on vlan-12 is br-vlan12. This host also has an IP address on the network itself so that I can access the VM host itself over the server subnet.

auto eth1 iface eth1 inet manual  auto br-vlan12   iface br-vlan12 inet static   address 10.10.2.21   network 10.10.2.0   netmask 255.255.255.0   broadcast 10.10.2.255   bridge_ports eth1.12   bridge_stp off 

One of the pieces that had me fumbling for so long is getting the bridge_ports and vlan_raw_interfaces specification on the bridge. It turns out when the former is specified in a certain way, the latter is not needed.

From interfaces(5):

VLAN AND BRIDGE INTERFACES To ease the configuration of VLAN interfaces, interfaces having . (full stop character) in the name are configured as 802.1q tagged virtual LAN interface. For exam‐ ple, interface eth0.1 is a virtual interface having eth0 as physical link, with VLAN ID 1.  For compatibility with bridge-utils package, if bridge_ports option is specified, VLAN interface configuration is not performed. 

I am still having some fun issues with routing on my network, where I can ping from my wifi network to the wired-network interface on my LAN, but not wifi -> server. I think this has to do with reverse path forwarding (RPF) checking on the server, given its default route is over the ‘wired’ network and not the server network interface. An invaluable tool to debugging these types of issues has been the sysctl setting below, logging martians. It logs instances where packets come into an interface which it does not expect, and therefore by-default rejects.

net.ipv4.conf.all.log_martians = 1 

The fun and breakage continues.

Helpful links found during my foray into this topic:

Unattended Ubuntu installs, part 2

In my initial post about unattended Ubuntu installs, I made the less-automated choice of hacking at the Ubuntu installation ISO and baking my preseed configuration right into the ISO. This proved to be incredibly inefficient and prevented a lot of the customization and quick-spin-up potential of what I interested in. In other words, if I wanted to spin up five identical VMs differing only by their hostname, was I really expected to bake five custom ISO’s whose preseed file only differed by their specification of the hostname?

Solution: With a bit of Internet poking, I found that you can specify the URL of a preseed file, accessible via HTTP, for your VM to read during OS installation as a kernel boot parameter. Given all this, there really was no reason to bake my own ISO in the first place. I had to test using virt-install specifying all these parameters on the command line, including using a straight Ubuntu install ISO. Results? Success!

For those curious of the command-line I used:

sudo /usr/bin/virt-install    --connect qemu:///system    --disk vol=<disk pool>/<disk volume>,cache=none    --extra-args "locale=en_US.UTF-8 console-keymaps-at/keymap=us console-setup/ask_detect=false console-setup/layoutcode=us keyboard-configuration/layout=USA keyboard-configuration/variant=US netcfg/get_hostname=<VM hostname> netcfg/get_domainname=<VM domain name> console=tty0 console=ttyS0,115200n8 preseed/url=<URL to preseed file>"    --location /mnt/raid1/dump/ubuntu-14.04.1-server-amd64.iso   --network bridge=<bridge if>   --name <VM name according to libvirt>    --os-type linux   --ram 512    --vcpus 1    --virt-type kvm  

Preseed file: This file can live on any accessible-from-your-VM http server. During the install process, it is retrieved via wget as part of the install procedure. But how do you specify the URL for the preseed file?

The only one modification I did have to make to my preseed file had to do with selecting a mirror. I was constantly prompted to select a mirror hostname. After another couple Google searches, I was left with what seems to work, by default picking a US-based HTTP mirror for Ubuntu packages:

d-i mirror/http/countries select US d-i mirror/http/hostname string archive.ubuntu.com d-i mirror/http/directory string /debian d-i mirror/http/mirror select us.archive.ubuntu.com d-i mirror/protocol select http d-i mirror/country string US d-i mirror/http/proxy string 

Enjoy!

Look ma’, no hands with Ubuntu installs.

In my day job, it’s all about automation. Automate what is repeatable, and move on to more interesting and not-yet-automated tasks. For a while, I’ve run a KVM/libvirt setup at home, running various iterations and distributions of Linux, OpenBSD and FreeBSD for various pet projects. Going through each distribution’s install procedure was getting old, requiring me to input the same parameters, set up the same users and passwords, over and over again. Given I use Ubuntu mostly as a VM guest, I dug into their preseed infrastructure, to be able to automate the installation and get me past the drudgery of adding another VM. Below are the steps and a bit of sample configuration that got me through the process.

I did find some examples of automating this all the way from virt-install (libvirt’s way of adding a VM instance to your cluster), but that is for another time.

[Update 2014-09-13: Even more unattended. Part 2]

Grab a Ubuntu Server ISO from their web site. Mount the ISO locally and rsync its contents to a new directory for your own customization.

mount -o loop /path/to/iso /some/mountpoint rsync -av /cdrom/ /opt/cd-image 

Now we’re left with the customization of the install. I wanted the installation to be completely hands-free. I shouldn’t have to enter in any partition information, user names, or network information. Right now my parameters are, for the most part, configured in the preseed file. My eventual goal is the factor those out into my own personal install script so that the command line arguments from my script are passed as kernel options to the install and are read at run-time as opposed to at CD-creation time. Doing it that way alleviates the need to re-create a new ISO with hard-coded values for the host name, domain name, and various network information, among others, each time you want to build a new VM in the preseed file.

/opt/cd-image/isolinux/txt.cfg (additions):

LABEL forman-preseed menu label ^Forman Preseed kernel /install/vmlinuz append preseed/file=/cdrom/preseed/ubuntu-server-custom.seed vga=788 initrd=/install/initrd.gz locale=en_US.UTF-8 console-keymaps-at/keymap=us console-setup/ask_detect=false console-setup/layoutcode=us keyboard-configuration/layout=USA keyboard-configuration/variant=USA -- 

/opt/cd-image/preseed/ubuntu-server-custom.preseed:

d-i debian-installer/locale string en_US d-i netcfg/choose_interface select eth0 d-i netcfg/get_hostname string preseedhost-1 d-i netcfg/get_domain string foobar.mylan d-i netcfg/wireless_wep string  d-i time/zone string US/Eastern d-i clock-setup/ntp boolean true  d-i partman/choose_partition select finish d-i partman/confirm boolean true d-i partman/confirm_nooverwrite boolean true d-i partman/default_filesystem string ext4 d-i partman-auto/init_automatically_partition select biggest_free d-i partman-auto/choose_recipe select atomic d-i partman-auto/method string regular d-i partman-auto/select_disk string /dev/vda d-i partman-md/confirm boolean true d-i partman-partitioning/confirm_write_new_label boolean true  d-i passwd/root-password-crypted password <echo "foo" | mkpasswd -m md5 -> d-i passwd/user-fullname string FirstName LastName d-i passwd/username string myfirstuser d-i passwd/user-password-crypted password <echo "foo" | mkpasswd -m md5 -> d-i user-setup/allow-password-weak boolean true d-i user-setup/encrypt-home boolean false  d-i mirror/http/proxy string d-i pkgsel/include string openssh-server irssi d-i pkgsel/upgrade select full-upgrade d-i pkgsel/update-policy select none  tasksel tasksel/first multiselect basic-ubuntu-server  d-i clock-setup/utc boolean true  d-i grub-installer/only_debian boolean true d-i grub-installer/timeout string 2 d-i finish-install/keep-consoles boolean true d-i finish-install/reboot_in_progress note 

This creates a VM with the various properties (highlights for brevity):

  • hostname = preseed-host1
  • domain name = foobar.mylan
  • network = DHCP via eth0
  • partitioning scheme = one big / partition, with space leftover for SWAP
  • one user with password set, along with root’s password.
  • install openssh-server and irssi to confirm package installation works.

Create the CD image:

IMAGE=custom.iso BUILD=/opt/cd-image/  mkisofs -r -V "Custom Ubuntu Install CD"    -cache-inodes    -J -l -b isolinux/isolinux.bin    -c isolinux/boot.cat    -no-emul-boot    -boot-load-size 4    -boot-info-table    -o $IMAGE    $BUILD 

Voila. Boot that as your install CD and behold the magic! Booting from this ISO inside your VM instance should leave you with a fully functionining instance.

The only real hiccups I hit along the way, given the multitude of documentation out there on the Internet, was getting past the keyboard selection prompts. Specifying the keyboard model, layout, and ‘ask_boolean false’ values for that set of questions inside the preseed file had no affect, I was still prompted. Those configuration values seemed to be REQUIRED to be set in the isolinux config, which in my case I stuck in txt.cfg.

Sources (of inspiration):

i3wm, i3bar, and rhythmbox

I was interested in customizing my i3wm setup a bit more, and wanted to display the current song playing in Rhythmbox while running the i3wm window manager.  It turned out to be just a few lines of configuration to my i3bar config.

First, I grabbed a copy of the Python wrapper around i3bar, wrapper.py. This wrapper merely takes the output of a command, wraps it in compliant JSON, and returns in a way that i3bar uses it generate its output. The wrapper allows the user to add on an arbitrary number of additions to their status bar, although you don’t want to overwhelm what is supposed to be just a status bar.

My own code I added to wrapper.py:

import subprocess  def get_rhythmbox_status():     """ Get the current song playing and elapsed time from rhythmbox if it is running. """     cmd = "rhythmbox-client --no-start --print-playing-format '%aa - %tt (%te)'"    try:        rb_output = subprocess.check_output(cmd, shell=True)        rb_output = rb_output.strip()    except subprocess.CalledProcessError, cpe:        rb_output = "Rhythmbox Client: Error"     if "(Unknown)" in rb_output:        rb_output = "Rhythmbox: Not Playing"     return rb_output 

This function executes rhythmbox-client. Command line flags include purposely not starting Rhythmbox if it is not started, and displaying a custom format of output related to the current song being played. This output is the artist name, track title, and elapsed time of the current track. If Rhythmbox is not started, no output is present in the i3bar. If Rhythmbox is started but no song is playing, “Rhythmbox: Not Playing” is displayed.

in the ‘main function’, I modified the line which inserts an entry in the JSON object:

j.insert(0, {'full_text' : get_rhythmbox_status(), 'name' : 'rhythmbox'}) 

Inside my i3wm config’s bar specification, my status_command line looks like the following:

status_command i3status --config ~/.i3/i3status.conf | ~/.i3/wrapper.py 

If your i3bar updates on an interval, you will see the current elapsed time of the song update as it plays.

FreeBSD on the desk, another try

After several years of mindlessly running Ubuntu on the desktop, I am attempting to dive (back) into running FreeBSD on the desktop. Considering that the majority of applications I use on the desktop are a browser (Firefox/Chrome), an ssh terminal, and Rhythmbox, how hard could this be?

Some of the hurdles

Given I still wanted to keep Ubuntu around and not redefine my default setup, I kept Grub2 as my bootloader on the MBR. I still needed a way to boot into FreeBSD at-will. I had installed FreeBSD on hd0a. Grub2 from Ubuntu makes finding the FreeBSD boot files incredibly easy:

>search -f /boot/kernel hd0, msdos2 > set root=(hd0, msdos2) > chainloader +1 > boot 

Considering it has been a while since I ran FreeBSD for anything serious, I had always debated between ports and packages. In my distant memory, packages were not built for every piece of software I wanted, and building ports has the downside of long compile times, and potentially hairy dependency issues. With FreeBSD 10.0-RELEASE, ‘pkg’ has become the default package manager, and thus far, I have had no issues finding packages for any software package I’ve wanted. Upon install, update the package repository:

# pkg update 

and use the various ‘pkg search’ and ‘pkg install’ variants to search for, and install the various applications.

I’ve always been curious in the various tiling window managers. i3wm seems to have the most-sane configuration structure among the various other tiling window managers (xmonad, awesomewm, etc). My i3wm configs up on Github.

I am still working on making the tiling-window manager thought process more second-nature. One instance I’m still attempting to wrap my head around, is when I fire up a full-screen window from Chrome, which ends up ‘under’ my main Chrome window. This makes the refrain ‘where the heck did my window go’ quite common. Alongside the fact that there is a lot of font configuration and xorg.conf hacking required to make the desktop what I consider ‘pretty.’

I find myself booting back into Ubuntu more often than not, given I’m more comfortable with the Unity window manager workflow. But I do boot into FreeBSD when time permits to try and hack on making it an actual usable desktop OS. The journey continues.

Wireless, now with more 802.11’s…

With nothing else to do around here tonight while the whole state is shut down thanks to a blizzard, I should catch up on some blog posts.

On my list of home network upgrades for the past several months was the wireless. As my wife and I add to our collection of smart phones, laptops, tablets, and wireless streaming devices (I am looking at you EOL Logitech Revue with Google TV) the amount of latency and available bandwidth started to show signs of strain. I had been running the wireless for several years on an Asus WL-500G Premium v2 router/wap, which only ran 802.11b/g over 2.4Ghz. It was time for an upgrade.

Welcome our new Asus RT-N66U 802.11b/g/n router/wap that handles dual channel 2.4Ghz and 5Ghz wifi.

I did some very unscientific comparisons before and after I performed the hardware upgrade. I pushed and pulled a ~763MB Ubuntu ISO across the wireless, through a 10/100Mb switch, via rsync over SSH from a server on the LAN. The following table shows rsync’s average speed and transfer duration from the point of view of a 15″ Macbook Pro connected via the wifi.

Old Wifi New Wifi 2.4Ghz New Wifi 5Ghz
Upload  1.87MB/s (6:46)  7.69MB/s (1:39)  5.57MB/s (2:17)
Download  2.61MB/s (4:51)  10.81MB/s (1:10)  10.39MB/s (1:13)

Needless to say, I am keeping the new router.

California 2012, thricely.

Condensed version of trip #3 to California.

San Diego

  • Sushi Ota: Sake and sushi with Mozilla folks. For the quality of the sushi (incredible), the price (reasonable) blew me away.
  • Tajima: Ramen! The spicy miso ramen here lives up to its name, be prepared.
  • Fish Market
  • Cucina Urbana: Serious Italian, and a wine list to match. Funky interior too.
  • Davanti Enoteca: Good tripe dish

San Francisco

  • State Bird Provisions: Dim sum delivery, California style.
  • Black Point Cafe: Coffee refuel near the Golden Gate Bridge. Killer latte. Gary Danko
  • Spruce: The duck. Yes, get the duck.
  • Philz Coffee: No justification needed. Had to restock the East Coast supply.
  • Acqurello
  • Izakaya Roku: More ramen! Sake!
  • Humphry Slocombe: Blue Bottle Vietnamese Coffee Ice Cream. Good lord. I can now die a happy man.
  • Incanto: Charcuterie zen master.

Napa/St Helena

  • White Rock Winery: We happened upon this one as they were dumping out the lower quality wine. So sad.
  • Oakville Grocery Co: Mid-Napa refuel. Bread, cheese, meats.
  • Saddleback Cellars
  • Gott’s Roadside: The juxtaposition of this in Napa is pretty jarring. Their milkshakes are a nice divider between all the wine.
  • Stag’s Leap Wine Cellars: This is where I cemented the fact that I like the smaller wineries, rather than the commercial behemoths.
  • Cliff Lede Winery
  • Morimoto Napa

Central California Coast

  • La Bicyclette (Carmel): Pizza, charcuterie, cheese. Take out. To be returned.
  • Sierra Mar at Post Ranch Inn (Big Sur): One of the most spectacular sunset views on the Pacific Coast. Food is damn tasty too.
  • Big Sur Bakery: Brunch doesn’t start until 1030am, but the fruit strudel and latte’s are killer.
  • Dover Canyon Winery: Our first foray into Paso Robles wines. Unexpectedly awesome. Along with their 185 lb. St Bernard named Thunder.
  • Turley Winery
  • Whalebone: Free chili with every tasting. Clutch with all the damn rain.
  • Adelaide Winery
  • Olavino: Olive oil and salt, one made with ghost chili. Holy crap. and holy good.
  • L’Adventure Winery

Nagios and Git hooks, a redux

A while back I blogged about how I hooked up Nagios and Git to run the Nagios preflight checks before restarting with a new checkin’s worth of configs. But the more I looked at how it all fit together, the more I knew it could be improved. A sed hack, expecting a certain pattern in the nagios.cfg? Bad bad bad. Most of the improvement revolves around Nagios’s ability to reference relative paths for its config files. Given the path of the ‘main’ nagios.cfg file, you can then reference directories that contain your services, hosts, and other custom commands, in relation to that main file. With this functionality I significantly improved the Git->Nagios pipeline.

First, the pre-receive hook

#!/bin/bash  umask 022  while read OLD_SHA1 NEW_SHA1 REFNAME;  do  export GIT_WORK_TREE=/tmp/nagiostest-$NEW_SHA1  mkdir -p $GIT_WORK_TREE /usr/bin/git checkout -f $NEW_SHA1 sudo /usr/sbin/nagios3 -v $GIT_WORK_TREE/nagios.cfg if [ "$?" -ne "0" ]; then  echo "Nagios Preflight Failed"  echo "See the above error, fix your config, and re-push to attempt to update Nagios."  exit 1  else  echo "Nagios Preflight Passed"  echo "Clearing temporary work directory."  rm -rf $GIT_WORK_TREE exit 0 fi done 

Using the GIT_WORK_TREE environment variable, which specifies Git’s working directory, I check out the new set of potential configs to a temporary directory. This provides a temporary ‘waiting room’ for the proposed configuration to be tested, before before being put into production. Imagine never (intentionally) breaking Nagios again because of a broken host or service specification. The main thing remember is that all references in the nagios.cfg to other config files (hosts, commands, etc) must be relative paths. I.E., I have lines that look like “cfg_dir=configs” in the nagios.cfg. Note the lack of absolute paths. We now run the Nagios pre-flight check (nagios -v) on the nagios.cfg in the Git work tree. Depending upon the exit value of ‘nagios -v’, 0 for success and 1 for failure, we either proceed or die immediately. If success, clean up our temporary run directory.

Now the post-receive hook:

#!/bin/sh  echo "Updating repo /etc/nagios3" sudo /usr/bin/update-gitrepo /etc/nagios3 

The post-receive hook merely runs a script, noted below, on the Nagios configuration directory.

Update-gitrepo:

#!/bin/sh umask 022  REPO_DIR=$1 cd ${REPO_DIR}  /usr/bin/git pull origin master 

Given the Git checkout’s directory, we fetch the most recent push to the repository.

For the final step we have to fix some permissions (given that my setup runs the repository through Gitolite as the git user). This hook is located in the actual checkout itself, /etc/nagios3, in the post-merge hook.

#!/bin/sh  sudo chown -R nagios:admin /etc/nagios3 sudo /etc/init.d/nagios3 restart 

A full commit and restart looks like this:

jforman@merlot:/mnt/raid1/personal/git/monitor/nagios/configs$ git push Counting objects: 7, done. Delta compression using up to 4 threads. Compressing objects: 100% (4/4), done. Writing objects: 100% (4/4), 414 bytes, done. Total 4 (delta 3), reused 0 (delta 0) remote: Previous HEAD position was c80fa03... turn off test notifications with notifications_enabled 0 remote: HEAD is now at f088dbc... Example: Add boilerplate header that file is managed by Git. remote: remote: Nagios Core 3.2.3 remote: Copyright (c) 2009-2010 Nagios Core Development Team and Community Contributors remote: Copyright (c) 1999-2009 Ethan Galstad remote: Last Modified: 10-03-2010 remote: License: GPL remote: remote: Website: http://www.nagios.org remote: Reading configuration data... remote: Read main config file okay... remote: Processing object config file '/tmp/nagiostest-f088dbcebf194edbce78068b6004cbbfca703432/commands.cfg'... remote: Processing object config directory '/etc/nagios-plugins/config'... remote: Processing object config file '/etc/nagios-plugins/config/ftp.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/mail.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/snmp_int.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/nt.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/http.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/real.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/check_nrpe.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/snmp_storage.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/disk.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/mysql.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/snmp_load.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/fping.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/dhcp.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/ssh.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/rpc-nfs.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/mailq.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/breeze.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/dummy.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/netware.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/hppjd.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/load.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/mrtg.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/apt.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/snmp_cpfw.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/snmp.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/snmp_process.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/snmp_env.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/news.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/ntp.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/telnet.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/users.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/snmp_mem.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/procs.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/ifstatus.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/games.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/disk-smb.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/tcp_udp.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/snmp_win.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/ping.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/pgsql.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/ldap.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/flexlm.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/dns.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/radius.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/snmp_vrrp.cfg'... remote: Processing object config directory '/tmp/nagiostest-f088dbcebf194edbce78068b6004cbbfca703432/configs'... remote: Processing object config file '/tmp/nagiostest-f088dbcebf194edbce78068b6004cbbfca703432/configs/timeperiods.cfg'... remote: Processing object config file '/tmp/nagiostest-f088dbcebf194edbce78068b6004cbbfca703432/configs/services.cfg'... remote: Processing object config file '/tmp/nagiostest-f088dbcebf194edbce78068b6004cbbfca703432/configs/commands.cfg'... remote: Processing object config file '/tmp/nagiostest-f088dbcebf194edbce78068b6004cbbfca703432/configs/hosts.cfg'... remote: Processing object config file '/tmp/nagiostest-f088dbcebf194edbce78068b6004cbbfca703432/configs/abstracts.cfg'... remote: Processing object config file '/tmp/nagiostest-f088dbcebf194edbce78068b6004cbbfca703432/configs/contacts.cfg'... remote: Read object config files okay... remote: remote: Running pre-flight check on configuration data... remote: remote: Checking services... remote: Checked 95 services. remote: Checking hosts... remote: Checked 10 hosts. remote: Checking host groups... remote: Checked 7 host groups. remote: Checking service groups... remote: Checked 0 service groups. remote: Checking contacts... remote: Checked 3 contacts. remote: Checking contact groups... remote: Checked 2 contact groups. remote: Checking service escalations... remote: Checked 0 service escalations. remote: Checking service dependencies... remote: Checked 56 service dependencies. remote: Checking host escalations... remote: Checked 0 host escalations. remote: Checking host dependencies... remote: Checked 0 host dependencies. remote: Checking commands... remote: Checked 181 commands. remote: Checking time periods... remote: Checked 4 time periods. remote: Checking for circular paths between hosts... remote: Checking for circular host and service dependencies... remote: Checking global event handlers... remote: Checking obsessive compulsive processor commands... remote: Checking misc settings... remote: remote: Total Warnings: 0 remote: Total Errors: 0 remote: remote: Things look okay - No serious problems were detected during the pre-flight check remote: Nagios Preflight Passed remote: Clearing temporary work directory. remote: Updating repo /etc/nagios3 remote: From monitor:nagios remote: * branch master -> FETCH_HEAD remote: Updating c80fa03..f088dbc remote: Fast-forward remote: configs/commands.cfg | 2 ++ remote: 1 file changed, 2 insertions(+) remote: * Restarting nagios3 monitoring daemon nagios3 remote: Waiting for nagios3 daemon to die.. remote: ...done. To git@monitor:nagios.git c80fa03..f088dbc master -> master 

Note that I do keep the Nagios package bundled commands in the /etc/nagios-plugins directory and have purposely not put those in the Git tree. This allows for updated Nagios packages from Ubuntu to update those commands accordingly without interfering with the Git repo.

Enjoy.