Third time’s a charm? Gitolite, Git, Nagios, and a bunch of hooks

I was hoping with my past posts on this topic, I would have enough examples to just copy-and-paste along to configure my Gitolite+Nagios monitoring setup. Not so true. It looked like there were semi-colon’s missing in my past examples. After looking at the huge number of changes in Gitolite, I had to re-do everything. Not to mention I always wanted a better way to manage the hooks as opposed to editing them directly on the host. In short, my goal is still simple: be able to manage and verify Nagios configuration remotely via Git. Below is how I did it. For the third time.

First, install Gitolite. I run Gitolite under the ‘git’ user on my Ubuntu-based VM, from now on called monitor1. I clone the Gitolite source under /home/git/gitolite.

In /home/git/.gitolite.rc, in the %RC block, uncomment:

LOCAL_CODE => "$rc{GL_ADMIN_BASE}/local", 

This option tells Gitolite we have some local configuration in our gitolite-admin repository under the ‘local’ directory. More on this later.

In the ENABLE list, uncomment:

'repo-specific-hooks', 

This option tells Gitolite we want to be able to to use repo-specific hooks, as opposed to having one set of hooks for all repositories.

Since several of our yet-to-be-defined hooks need elevated permissions, I have configured a sudoers file to allow so.

%nagios-admin ALL=(ALL) NOPASSWD: /usr/sbin/nagios3 -v /tmp/nagiostest-*/nagios.cfg %nagios-admin ALL=(ALL) NOPASSWD: /usr/sbin/service nagios3 restart 

Our ‘nagios’ user is added to the nagios-admin group, along with the git user. This allows us via Gitolite’s hooks to test, update, and restart the Nagios installation.

This concludes all the work on monitor1 as it relates to Gitolite.

On your local workstation, clone the gitolite-admin repo. I’ve chosen to name the repo containing my Nagios configuration, ‘nagios’. At this point, it is probably safe to copy a known-working copy of your Nagios configuration files into the nagios repository itself. The steps following here, if done in one fell swoop, could completely blow away your /etc/nagios3 directory on monitor1 if you are not careful.

One modification necessary for the nagios.cfg itself is to modify the references to the path of the the configuration files. By default, the nagios.cfg lists an absolute path to the files, e.x: /etc/nagios3/conf.d/. In our case, we will be checking out the configuration files to a temporary directory while we run our pre-flight checks and need to use a relative path instead to make this possible.

Therefore in your nagios.cfg file, perform the following changes:

cfg_file=commands.cfg cfg_dir=conf.d 

Now that I look at it, I’m not quite sure why these are specified separately, as your commands.cfg file could live under conf.d. But I’ll leave that for readers who have their own structure preferences. The key here is that relative paths must be used, NOT absolute ones.

Next we move onto the gitolite-admin configuration:

repo nagios   RW+ = jforman_rsa   option hook.pre-receive = nagios-pre-receive   option hook.post-receive = nagios-post-receive   option hook.post-update = nagios-post-update 

This tells Gitolite the name of my nagios config repository, who is ACL’d to read and write to it, and which hooks I wish to override with my custom hooks. Note that in Gitolite, you can only override these three hooks, pre and post-receive, and post-update. Other hooks such a post-merge and merge are special to Gitolite and you will be returned an error if you attempt to override them. Note that each hook, nagios-pre-receive, and so on, corresponds to a file name that will live under my gitolite-admin repository under the ‘local’ directory.

Now we come to the point of defining our hooks. Under your gitolite-admin directory, create the directory structure ‘hooks/repo-specific’ under the directory we defined in the above LOCAL_CODE definition. In our case, that corresponds to ‘local’.

In other words. in our local checkout of the gitolite-admin repository:

mkdir -p ${gitolite_admin_path}/local/hooks/repo-specific 

Under this repo-specific directory, using whatever language you prefer (Python, Shell, Perl, etc), create the files for the repository’s hooks.

gitolite-admin$ tree local/ local/   └── hooks     └── repo-specific       ├── nagios-post-receive       ├── nagios-post-update       └── nagios-pre-receive 

nagios-pre-receive:

#!/bin/bash  umask 022  while read OLD_SHA1 NEW_SHA1 REFNAME;  do  export GIT_WORK_TREE=/tmp/nagiostest-$NEW_SHA1  mkdir -p $GIT_WORK_TREE /usr/bin/git checkout -f $NEW_SHA1 sudo /usr/sbin/nagios3 -v $GIT_WORK_TREE/nagios.cfg if [ "$?" -ne "0" ]; then   echo "Nagios Preflight Failed"    echo "See the above error, fix your config, and re-push to attempt to update Nagios."    exit 1  else    echo "Nagios Preflight Passed"    echo "Clearing temporary work directory."    rm -rf $GIT_WORK_TREE exit 0 fi done 

nagios-pre-receive: Using the most recent commit, checkout this body of work into a temp directory and run the Nagios pre-flight checks over it. If those checks pass, exit 0 (without error). If those pre-flight checks fail, error out. This latter case will stop the Git push completely. Your running Nagios configs in /etc/nagios3 are untouched.

nagios-post-recieve

#!/bin/bash  echo "Updating repo /etc/nagios3" /usr/bin/update-gitrepo /etc/nagios3 

nagios-post-receive: This runs a companion script to update the cloned git repository at /etc/nagios3. Note that this is only run if the post-receive succeeds, and executes after the merge step of the git push has succeeded, which means our Gitolite nagios repository has now merged the commits we are attempting to push.

nagios-post-update

#!/bin/bash  sudo chown -R root:nagios-admin /etc/nagios3 sudo /usr/sbin/service nagios3 restart 

nagios-post-update: The post-update step runs after the post-receive, ensuring our permissions are correct and then restarts Nagios.

At this point, the custom hooks and gitolite.conf should be committed and pushed to the remote Gitolite gitolite-admin repository. No more storing hooks in the bare repository itself! The hooks themselves are version controlled. This was what bugged me the most about my prior solutions. I always hated not having any history on how I fixed (broke) the scripts in the past.

update-gitrepo

#!/bin/bash umask 022  REPO_DIR=$1 cd ${REPO_DIR} unset GIT_DIR /usr/bin/git pull origin master 

update-gitrepo: Lives on the monitor1 under /usr/bin, and merely executes a ‘git pull’ under the passed directory.

This last bit of configuration I’m not too happy with, given it feels so manual and hacky. It’s how you get /etc/nagios3 to be the Git repository checkout itself. On monitor1, I normally do this initial work in /tmp, as the ‘git’ user.

cd /tmp git clone /home/git/repositories/nagios.git/ (as root) mv /etc/nagios3 /etc/nagios3.notgit (as root) mv /tmp/nagios /etc/nagios3 (as root) chown -R root:nagios-admin /etc/nagios3 /usr/bin/update-gitrepo /etc/nagios3 

This performs a checkout of the Nagios repository itself (note that we’re skirting Gitolite’s ACL control and accessing the repository directly on the file system). If you wish to further control who can check out the nagios directory, create an ssh-key for your ‘git’ user locally, add to the gitolite-admin repository ACL, and perform your checkout over SSH as opposed to using the absolute-path of the directory.

Wow. This post turned out to be much longer than I expected. If you’ve made it this far, you should have the groundwork laid to do remote clones of your Nagios configuration files, and have the ability to run the pre-flight check verify their correctness before ever getting near your production Nagios directory. Good luck.