Nagios and Git hooks, a redux

A while back I blogged about how I hooked up Nagios and Git to run the Nagios preflight checks before restarting with a new checkin’s worth of configs. But the more I looked at how it all fit together, the more I knew it could be improved. A sed hack, expecting a certain pattern in the nagios.cfg? Bad bad bad. Most of the improvement revolves around Nagios’s ability to reference relative paths for its config files. Given the path of the ‘main’ nagios.cfg file, you can then reference directories that contain your services, hosts, and other custom commands, in relation to that main file. With this functionality I significantly improved the Git->Nagios pipeline.

First, the pre-receive hook

#!/bin/bash  umask 022  while read OLD_SHA1 NEW_SHA1 REFNAME;  do  export GIT_WORK_TREE=/tmp/nagiostest-$NEW_SHA1  mkdir -p $GIT_WORK_TREE /usr/bin/git checkout -f $NEW_SHA1 sudo /usr/sbin/nagios3 -v $GIT_WORK_TREE/nagios.cfg if [ "$?" -ne "0" ]; then  echo "Nagios Preflight Failed"  echo "See the above error, fix your config, and re-push to attempt to update Nagios."  exit 1  else  echo "Nagios Preflight Passed"  echo "Clearing temporary work directory."  rm -rf $GIT_WORK_TREE exit 0 fi done 

Using the GIT_WORK_TREE environment variable, which specifies Git’s working directory, I check out the new set of potential configs to a temporary directory. This provides a temporary ‘waiting room’ for the proposed configuration to be tested, before before being put into production. Imagine never (intentionally) breaking Nagios again because of a broken host or service specification. The main thing remember is that all references in the nagios.cfg to other config files (hosts, commands, etc) must be relative paths. I.E., I have lines that look like “cfg_dir=configs” in the nagios.cfg. Note the lack of absolute paths. We now run the Nagios pre-flight check (nagios -v) on the nagios.cfg in the Git work tree. Depending upon the exit value of ‘nagios -v’, 0 for success and 1 for failure, we either proceed or die immediately. If success, clean up our temporary run directory.

Now the post-receive hook:

#!/bin/sh  echo "Updating repo /etc/nagios3" sudo /usr/bin/update-gitrepo /etc/nagios3 

The post-receive hook merely runs a script, noted below, on the Nagios configuration directory.

Update-gitrepo:

#!/bin/sh umask 022  REPO_DIR=$1 cd ${REPO_DIR}  /usr/bin/git pull origin master 

Given the Git checkout’s directory, we fetch the most recent push to the repository.

For the final step we have to fix some permissions (given that my setup runs the repository through Gitolite as the git user). This hook is located in the actual checkout itself, /etc/nagios3, in the post-merge hook.

#!/bin/sh  sudo chown -R nagios:admin /etc/nagios3 sudo /etc/init.d/nagios3 restart 

A full commit and restart looks like this:

jforman@merlot:/mnt/raid1/personal/git/monitor/nagios/configs$ git push Counting objects: 7, done. Delta compression using up to 4 threads. Compressing objects: 100% (4/4), done. Writing objects: 100% (4/4), 414 bytes, done. Total 4 (delta 3), reused 0 (delta 0) remote: Previous HEAD position was c80fa03... turn off test notifications with notifications_enabled 0 remote: HEAD is now at f088dbc... Example: Add boilerplate header that file is managed by Git. remote: remote: Nagios Core 3.2.3 remote: Copyright (c) 2009-2010 Nagios Core Development Team and Community Contributors remote: Copyright (c) 1999-2009 Ethan Galstad remote: Last Modified: 10-03-2010 remote: License: GPL remote: remote: Website: http://www.nagios.org remote: Reading configuration data... remote: Read main config file okay... remote: Processing object config file '/tmp/nagiostest-f088dbcebf194edbce78068b6004cbbfca703432/commands.cfg'... remote: Processing object config directory '/etc/nagios-plugins/config'... remote: Processing object config file '/etc/nagios-plugins/config/ftp.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/mail.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/snmp_int.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/nt.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/http.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/real.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/check_nrpe.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/snmp_storage.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/disk.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/mysql.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/snmp_load.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/fping.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/dhcp.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/ssh.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/rpc-nfs.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/mailq.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/breeze.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/dummy.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/netware.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/hppjd.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/load.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/mrtg.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/apt.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/snmp_cpfw.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/snmp.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/snmp_process.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/snmp_env.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/news.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/ntp.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/telnet.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/users.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/snmp_mem.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/procs.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/ifstatus.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/games.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/disk-smb.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/tcp_udp.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/snmp_win.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/ping.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/pgsql.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/ldap.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/flexlm.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/dns.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/radius.cfg'... remote: Processing object config file '/etc/nagios-plugins/config/snmp_vrrp.cfg'... remote: Processing object config directory '/tmp/nagiostest-f088dbcebf194edbce78068b6004cbbfca703432/configs'... remote: Processing object config file '/tmp/nagiostest-f088dbcebf194edbce78068b6004cbbfca703432/configs/timeperiods.cfg'... remote: Processing object config file '/tmp/nagiostest-f088dbcebf194edbce78068b6004cbbfca703432/configs/services.cfg'... remote: Processing object config file '/tmp/nagiostest-f088dbcebf194edbce78068b6004cbbfca703432/configs/commands.cfg'... remote: Processing object config file '/tmp/nagiostest-f088dbcebf194edbce78068b6004cbbfca703432/configs/hosts.cfg'... remote: Processing object config file '/tmp/nagiostest-f088dbcebf194edbce78068b6004cbbfca703432/configs/abstracts.cfg'... remote: Processing object config file '/tmp/nagiostest-f088dbcebf194edbce78068b6004cbbfca703432/configs/contacts.cfg'... remote: Read object config files okay... remote: remote: Running pre-flight check on configuration data... remote: remote: Checking services... remote: Checked 95 services. remote: Checking hosts... remote: Checked 10 hosts. remote: Checking host groups... remote: Checked 7 host groups. remote: Checking service groups... remote: Checked 0 service groups. remote: Checking contacts... remote: Checked 3 contacts. remote: Checking contact groups... remote: Checked 2 contact groups. remote: Checking service escalations... remote: Checked 0 service escalations. remote: Checking service dependencies... remote: Checked 56 service dependencies. remote: Checking host escalations... remote: Checked 0 host escalations. remote: Checking host dependencies... remote: Checked 0 host dependencies. remote: Checking commands... remote: Checked 181 commands. remote: Checking time periods... remote: Checked 4 time periods. remote: Checking for circular paths between hosts... remote: Checking for circular host and service dependencies... remote: Checking global event handlers... remote: Checking obsessive compulsive processor commands... remote: Checking misc settings... remote: remote: Total Warnings: 0 remote: Total Errors: 0 remote: remote: Things look okay - No serious problems were detected during the pre-flight check remote: Nagios Preflight Passed remote: Clearing temporary work directory. remote: Updating repo /etc/nagios3 remote: From monitor:nagios remote: * branch master -> FETCH_HEAD remote: Updating c80fa03..f088dbc remote: Fast-forward remote: configs/commands.cfg | 2 ++ remote: 1 file changed, 2 insertions(+) remote: * Restarting nagios3 monitoring daemon nagios3 remote: Waiting for nagios3 daemon to die.. remote: ...done. To git@monitor:nagios.git c80fa03..f088dbc master -> master 

Note that I do keep the Nagios package bundled commands in the /etc/nagios-plugins directory and have purposely not put those in the Git tree. This allows for updated Nagios packages from Ubuntu to update those commands accordingly without interfering with the Git repo.

Enjoy.