At work we have a monitoring configuration workflow where our Nagios config files are parsed and generated before they are allowed to be ‘svn commit’ed. I know this verification has saved me many times when trying to add new hosts or services, since everything might not be ready for prime time. I wanted to see if I could recreate this scenario at home using Git hooks, if only for my own interest and curiosity.
My proposed workflow: Start with a local clone of the Nagios configs. Add a host, service, or contact to a local nagios Config file. Git commit locally. Git push to the remote repo on the monitoring box. The Nagios config parser would run remotely verifying the syntactic-correctness of the proposed-check in. If these checks fail, reject the push. If parsing succeeds, accept the push, and restart Nagios (hence reloading the new configs). I wanted to alleviate the need for an admin to log into the monitoring machine at all. This would eliminate the need to add sudo permissions to a group of users to allow them to restart the Nagios service.
Without further delay, on to the code. The first piece, was the pre-receive hook with Git. I used this hook specifically because it has the power to accept or reject a push.
Behold, $git_repo/.hooks/pre-receive:
#!/bin/bash
while read OLD_SHA1 NEW_SHA1 REFNAME; do
export GIT_WORK_TREE=/tmp/nagiosworkdir
/usr/bin/git checkout -f $NEW_SHA1
sed -i "s|cfg_dir=CHANGEWITHGIT|cfg_dir=${GIT_WORK_TREE}\/config|g" $GIT_WORK_TREE/nagios.cfg
sudo -u root /bin/chgrp -R nagios $GIT_WORK_TREE
sudo -u root /usr/sbin/nagios3 -v $GIT_WORK_TREE/nagios.cfg > $GIT_WORK_TREE/check.out
NAGIOS_CHECK_STATUS=$?
echo "Nagios Config Check Exit Status:" $NAGIOS_CHECK_STATUS
if [ "$NAGIOS_CHECK_STATUS" -ne 0 ]
then
echo "Your configs did not parse correctly, there was an error. Output follows."
cat $GIT_WORK_TREE/check.out
exit 1
else
echo "Your configs look good and parsed correctly."
exit 0
fi
done
The operations here flow from checking out the pending push to a ‘temporary directory’ (explained below), and then making a slight modification to the nagios.cfg to handle our temp configs being somewhere other than the Nagios install directory. This is needed since we want to parse our configs, and not the ones currently in-use on the machine. After this is done, we parse the output of that check, and based upon that output, either accept or reject the push. In the case of a rejection, the output of the verification is printed for the user to see. This output is pretty clear when explaining where the error is.
And now the post-receive:
#!/bin/bash
cd /etc/nagios3/testing
/usr/bin/env -i /usr/bin/git pull
echo "Restarting Nagios now"
sudo -u root /usr/sbin/service nagios3 restart
The post-receive happens after the entire process is done [1]. We change into the directory of the Git clone of your monitoring config repository. which itself is inside /etc/nagios3, and execute a ‘git pull.’ The need to wrap it around ‘env’ is because of an issue I also learned in this process. [2] The GIT_DIR directory is set to the directory of your repository. In this case, we must change to another directory on this machine, which is outside of our initial Git repo. Therefore we must ‘unset’ this variable, which is what the ‘env’ execution does.
This is pretty straightforward for the most part. The part of the process new to me was the GIT_WORK_TREE. I received a a pretty simple explanation from the author of Gitolite. He explained that the work tree directory is a directory where the potential new commit can be checked out before it is actually allowed to be pushed into the remote repository. Essentially, the opposite of a bare repository. The reason I do this is because I want to do the parsing check before I actually allow the push. I can’t ‘git pull’ the new repository directly into the Nagios configuration directory before it is actually committed. GIT_WORK_TREE allows this intermediary functionality.
The reasoning for the ‘sed’ modification in that file is that since I will be checking-out the intermediary files into a new directory, I need a Nagios config file that references this intermediary directory. How did I get around this? I copied the nagios.cfg file I would run my install with into the repo, and changed the cfg_dir directory to something I can easily modify during the check in (cfg_dir=CHANGEWITHGIT). This file is not expected to be used on the actual server, so do not copy this one over your actual Nagios.cfg file.
The chgrp magic is because the user who runs the Nagios application must have read access to the configuration files.
Relevant sudoers lines needed:
git ALL=(ALL) NOPASSWD: /bin/chgrp -R nagios /tmp/nagiosworkdir
git ALL=(ALL) NOPASSWD: /usr/sbin/nagios3 -v /tmp/nagiosworkdir/nagios.cfg
git ALL=(ALL) NOPASSWD: /usr/sbin/service nagios3 restart
The three lines above are as such. (1) Allow the git user to, without a password, change the group of the temporary Nagios work dir. (2) Allow the git user to, without password, run the verification of the to-be-committed files. (3) Actually restart the Nagios service.
A successful commit, including a restart of Nagios looks like the following:
jforman@merlot:~/devel/testing/config$ git commit . && git push
[master 2f796e3] testing for the blog post
1 files changed, 1 insertions(+), 1 deletions(-)
Counting objects: 7, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (4/4), done.
Writing objects: 100% (4/4), 394 bytes, done.
Total 4 (delta 2), reused 0 (delta 0)
remote: Previous HEAD position was ca6dd74... fix now
remote: HEAD is now at 2f796e3... testing for the blog post
remote: Nagios Config Check Exit Status: 0
remote: Your configs look good and parsed correctly.
remote: From /home/git/repositories/testing
remote: ca6dd74..2f796e3 master -> origin/master
remote: Updating ca6dd74..2f796e3
remote: Fast-forward
remote: config/contacts_nagios2.cfg | 2 +-
remote: 1 files changed, 1 insertions(+), 1 deletions(-)
remote: Restarting Nagios now
remote: * Restarting nagios3 monitoring daemon nagios3
remote:
remote: ...done.
To git@monitor:testing.git
ca6dd74..2f796e3 master -> master
This was definitely an eye-opening learning process for me into all the moving parts of Git. I hope this Howto helps those out there looking for a solution like this. Enjoy!
Addendum:
For those wondering about the various permissions of the git_work_tree directory and also the repo checkout inside /etc/Nagios3:
monitor:/tmp/nagiosworkdir
drwxrwxr-x 3 git nagios 4096 2010-11-16 16:13 nagiosworkdir
monitor:/etc/nagios3/testing
drwxr-xr-x 4 git nagios 4096 2010-11-15 08:36 testing
[1] http://progit.org/book/ch7-3.html
[2] http://debuggable.com/posts/git-tip-auto-update-working-tree-via-post-receive-hook:49551efe-6414-4e86-aec6-544f4834cda3
Jeff,
I have to admit, this is great so far!