To make this setup work, first you’ll need a virtual IP to point puppet.machine.com at and setup keepalived to manage it. The keepalived setup was fairly straight forward, both machines were on the same network so I just picked an IP and configured keepalived on each machine. In my case the configurations were as follows:

vrrp_instance VI_1 {
        interface bond0
        state MASTER
        virtual_router_id 51
        priority 101
        authentication {
            auth_type PASS
            auth_pass PASSWORD
        }
        virtual_ipaddress {
                192.168.33.61/26 dev bond0
        }
        notify_master /sw/keepalived/scripts/notify_master.sh
        notify_backup /sw/keepalived/scripts/notify_backup.sh
        notify_fault /sw/keepalived/scripts/notify_backup.sh
}

Basically for bonded interface 0, add the virtual interface of 192.168.33.61/26, and this machine has a priority of 101, or master. The three entries at the bottom are what to run if the machine should fail over or fail back. The other machine has a similar setup, except it has the priority of 100, or backup.

After starting up keepalived on both machines, they negotiate and the machine with the higher priority ends up with the IP enabled on bond0 as a secondary address.

Next I needed something to manage fail-over of the puppet “service” should the interface/IP stay up and just the puppet service fail. For this I decided to use HAProxy. It’s configuration is, again, fairly straight forward. You need to setup a front end to listen on the default puppet master port, and a back end that has both your puppet master machines in it. The configuration is as follows:

global
    log         127.0.0.1 local2
    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    maxconn     10000
    user        haproxy
    group       haproxy
    daemon
    stats socket /var/lib/haproxy/stats

defaults
    log                     global
    option                  redispatch
    retries                 3
    timeout http-request    10s
    timeout queue           1m
    timeout connect         10s
    timeout client          1m
    timeout server          1m
    timeout http-keep-alive 10s
    timeout check           10s
    maxconn                 8000

frontend        puppet 192.168.33.61:8140
    mode tcp
    default_backend     puppet0

backend puppet0
    mode        tcp
    option      ssl-hello-chk
    balance     roundrobin
    server      server1 192.168.33.17:8140 check
    server      server2 192.168.33.27:8140 check backup

This configuration and the installation of haproxy is mirrored on the second server.

Now if you recall I had some scripts that were to run should keepalived feel the need to fail to/from the backup server. Because haproxy binds to the address on start up, you can’t have it running on your backup server. To work around the script is run upon keepalived starting that IP on the system, the script starts haproxy.

With the complicated part done, it was just a matter of mirroring my puppet manifests and files between the two machines, and making sure they stay up to date. Ideally you’d have them on some sort of NAS which is just mounted on both machines which would make them being out of date impossible.

You just need to:

1. Copy over ‘puppet_dashboard.rb’ with a modified HOST line to the other host.
2. Modify /etc/sysconfig/puppetmaster.
3. Turn on reporting on the clients.
4. Restart puppetmaster.

Since the puppet_dashboard.rb is just making an HTTP post to the dashboard server, it can be coming from anywhere.

My next trick is useful for getting around having your HTTP usage tracked and filtered, this is just taking advantage of the “-D” option within the ‘ssh’ program.  When you invoke the -D option, followed with a port number you not only connect to the remote machine via SSH, but you also create a SOCKS proxy server on the port specified.  Setting up your browser’s proxy settings to point at that port on your local machine will securely forward any requests over to the remote machine, where they will then go out to their final destination.

When you combine corkscrew with ssh -D, you end up with a single, secure connection through the HTTP proxy to your remote machine with a SOCKS proxy through it.  This effectively gives you an easy unfiltered hole through to the outside.

You can also use this trick for getting rsync through a http firewall that might otherwise block it.  To accomplish this you just have setup your RSYNC_CONNECT_PROG environment variable to “/path/to/corkscrew <proxy> <proxy_port> %H 873 <auth_file>”.

Disclaimer: Check with your IT policies before attempting any of the above to make sure its allowed!

I’ll start with the tripwire.pp file for puppet, in this file you’ll define your tripwire class and associated files and packages:

class tripwire {
        file { "/etc/tripwire/Makefile":
                owner => root,
                group => root,
                mode => 440,
                source => "puppet:///etc/tripwire/Makefile",
                require => Package[tripwire],
        }
        file { "/etc/tripwire/site.key":
                owner => root,
                group => root,
                mode => 440,
                source => "puppet:///etc/tripwire/site.key",
                require => Package[tripwire],
        }
        file { "/etc/tripwire/twcfg.txt":
                owner => root,
                group => root,
                mode => 440,
                source => "puppet:///etc/tripwire/twcfg.txt",
                require => Package[tripwire],
        }
        file { "/etc/tripwire/twpol.txt":
                owner => root,
                group => root,
                mode => 440,
                source => "puppet:///etc/tripwire/twpol.txt",
                require => Package[tripwire],
        }

        file { "/etc/cron.daily/tripwire-check":
                owner => root,
                group => root,
                mode => 755,
                source => "puppet:///etc/cron.daily/tripwire-check",
                require => Package[tripwire],
        }

        file { "/var/lib/tripwire/tripwire.twd":
                owner => root,
                group => root,
                mode => 440,
        }

        package {
                "tripwire":
                        ensure => "installed",
        }

        exec {
                "make -C /etc/tripwire":
                        cwd => "/etc/tripwire",
                        path => ["/bin", "/usr/bin" ],
                        creates => "/var/lib/tripwire/tripwire.twd",
                        timeout => 10000
        }

        exec {
                "tripwire --update-policy -Z low --local-passphrase '' --site-passphrase '' --quiet /etc/tripwire/twpol.txt":
                        cwd => "/etc/tripwire",
                        path => ["/sbin", "/usr/sbin" ],
                        timeout => 10000,
                        refreshonly => "true",
                        subscribe => File["/etc/tripwire/twpol.txt"],

        }
}

You’ll notice there is a Makefile defined, this will make remote management much simpler, it contains some simple house keeping items. I do not use all these within my puppet implementation, however:

all: tw.cfg tw.pol /var/lib/tripwire/tripwire.twd

clean:
        /bin/rm -f /etc/tripwire/local.key /etc/tripwire/tw.cfg /etc/tripwire/tw.pol /var/lib/tripwire/tripwire.twd

init: clean /var/lib/tripwire/tripwire.twd

local.key:
        /usr/sbin/twadmin --generate-keys --local-keyfile local.key --local-passphrase ''

tw.cfg: site.key twcfg.txt
        /usr/sbin/twadmin --create-cfgfile --site-keyfile site.key --site-passphrase '' twcfg.txt

tw.pol: tw.cfg twpol.txt
        if [ -f tw.pol -a -f /var/lib/tripwire/tripwire.twd ]; \
        then /usr/sbin/tripwire --update-policy --local-passphrase '' --quiet --secure-mode low --site-passphrase '' twpol.txt; \
        else /usr/sbin/twadmin --create-polfile --site-passphrase '' twpol.txt; \
        fi

/var/lib/tripwire/tripwire.twd: local.key site.key
        /usr/sbin/tripwire --init --local-passphrase ''

My twcfg.txt is very simple:

# required
DBFILE                 = /var/lib/tripwire/tripwire.twd
LOCALKEYFILE           = /etc/tripwire/local.key
POLFILE                = /etc/tripwire/tw.pol
REPORTFILE             = /var/lib/tripwire/report/$(DATE).twr
SITEKEYFILE            = /etc/tripwire/site.key

# e-mail notification
EMAILREPORTLEVEL       = 3
GLOBALEMAIL            = email@domain.tld
MAILMETHOD             = SENDMAIL
MAILNOVIOLATIONS       = false
MAILPROGRAM            = /usr/sbin/sendmail -oi -t

# other
EDITOR                 = /usr/bin/vim
LATEPROMPTING          = false
LOOSEDIRECTORYCHECKING = false
REPORTLEVEL            = 4
SYSLOGREPORTING        = true
TEMPDIRECTORY          = /var/tmp

Other than including the tripwire class within your site.pp file, thats about it.

Credit: The makefile was created by Steve Coile.

To do this you’ll need to copy over the following binarys from the server installation into your client’s bin directory:

• bbtest-net
• hobbitping

You’ll also need a copy of your bb-hosts and bb-services file, you’ll need to place these in your client’s etc folder. In your bb-hosts file you’ll only need the hosts that are within the private network you intend to ping, you really don’t need any grouping information within it either since when the data is sent in to your xymon server it will be assigned to the correct hosts there.

At this point you need to add a section for bbnet within your clientlaunch.cfg, yours may be different since your file locations my vary, so edit to fit your situation:

[bbnet]
ENVFILE /usr/lib/hobbit/client/etc/clientlaunch.cfg
CMD /usr/lib/hobbit/client/bin/bbtest-net --report --ping --checkresponse
LOGFILE $BBSERVERLOGS/bb-network.log
INTERVAL 5m

Since bbtest-net needs to know what to do you’ll need to copy its configuration section from hobbitserver.cfg to hobbitclient.cfg:

# For bbtest-net
CONNTEST="TRUE"
IPTEST_2_CLEAR_ON_FAILED_CONN="TRUE"
NONETPAGE=""
FPING="/usr/sbin/fping"
NTPDATE="ntpdate"
TRACEROUTE="traceroute"
BBROUTERTEXT="router"
NETFAILTEXT="not OK"

And there you have it – restart your client and it should be able to ping the clients within the private network and send to your Xymon server.  You’ll notice a column for bbtest show up for the client you setup to ping as well.   This should work for all network tests, however I’ve just tested ping.

Total Slots: 2
Total Used Slots: 2

Queue: MainQueue
Slots: 2
Reserved Slots: 0
Used Slots: 2

The master is the same, but contains information for the entire cluster. When all slots are taken, the test icon/color turns to ‘green’, when some are available the test icon/color changes to ‘clear’ I’ve also created an easy to understand graph which consists of a green area that is the total number of slots, which a red area over it showing the used slots. As slots free up the green shows itself noting there are slots available.

The Script is available for download here.

The Graph definitions are:

[sge]
TITLE SGE Used Slots
YAXIS # Used Slots
DEF:SlotsUsed=sge.rrd:TotalUsedSlots:AVERAGE
DEF:SlotsTotal=sge.rrd:TotalSlots:AVERAGE
AREA:SlotsTotal#00CC00:Total Slots
AREA:SlotsUsed#FF0000:Used Slots
COMMENT:\n
GPRINT:SlotsUsed:LAST:Used Slots  : %5.1lf (cur)
GPRINT:SlotsUsed:MAX: : %5.1lf (max)
GPRINT:SlotsUsed:MIN: : %5.1lf (min)
GPRINT:SlotsUsed:AVERAGE: : %5.1lf (avg)\n
GPRINT:SlotsTotal:LAST:Total Slots   : %5.1lf (cur)
GPRINT:SlotsTotal:MAX: : %5.1lf (max)
GPRINT:SlotsTotal:MIN: : %5.1lf (min)
GPRINT:SlotsTotal:AVERAGE: : %5.1lf (avg)\n

Update: I’ve modified my script to NOT change colors.  Seems xymon really ignores whatever you sent in since it really thinks you have no data when you turn to ‘clear’ – it was breaking my graphs.

Since I’ve had to write quite a few of these I thought I’d share a simple recipe for getting started with these scripts.   Most all our scripts are written in Perl, its just what I’m used too and what most people in our company is fluent in.   Our older scripts (from when we actually ran Big Brother) are almost all shell scripts and I have written at least one in Python since that particular service’s library was in python.   For this example I’m going to use Perl default template:

#!/usr/bin/perl -w
#############################################################################
# $Id: $
#############################################################################
use strict;

## BB and related test constants
#############################################################################

use constant GREEN => 'green';
use constant YELLOW => 'yellow';
use constant RED => 'red';

## BB Global variables
#############################################################################

my $prev_run_log = "$ENV{BBTMP}/$ENV{MACHINE}.lighttpd.data";
my $bbtest = 'testname';
my $color = GREEN;
my $status = $bbtest . " OK";

## Main Program
#############################################################################
{

my $DATA = "";

## TEST STUFF HERE

## Send to Hobbit
#############################################################################
my $report_date = `/bin/date`;
chomp($report_date);

system("$ENV{BB} $ENV{BBDISP} 'status $ENV{MACHINE}.$bbtest $color $report_date - $status\n\n$DATA'\n");

}

This script by itself will send a test called ‘testname’, which is green and has the status of ‘OK’.  Since we have not defined anything to test it won’t contain any data with it.  To run this script you need to be within Xymon’s “shell”, which is basically sh, with some enviroment variables set.  To get there, go to ’bin’ directory within your hobbit client installation, in my case this is: “/home/hobbit/client/bin”, you’ll need to run ‘bbcmd’, this will drop you to a prompt which looks something like this:

[rar@apollo bin]$ ./bbcmd
2009-05-09 10:10:35 Using default environment file /home/hobbit/client/etc/hobbitclient.cfg
sh-3.2$

Lets change the test name to something else and gather some data from the system we are running on.   To change the test name to say ‘modules’, just change the $bbtest to ‘modules’.   How about testing what kernel modules are loaded – or basically just parse the output of ‘lsmod’.   To do this we’d need to capture the output of the command, this is easy enough in perl, something like this would work just fine:

my $lsmod = `/sbin/lsmod`;

With the output of ’lsmod’ captured we can do whatever we want with it.  Lets say we just want the modules, and no other info about them in an array:

my $lsmod = `/sbin/lsmod`;
my @modules = split(/\n/, $lsmod);
my @module_names;

foreach my $module (@modules){
my ($name, $junk) = split(/\s.+/, $module);
push @module_names, $name;
}

Now that we have our array created that we can easily loop through for tests, lets run a test on them to see if we have loaded any modules for ‘ext’ file systems:

foreach my $module (@module_names) {
if ($module =~ "ext"){
$color = RED;
$status = $bbtest . " NOT OK"
}
}

OK,  since we’d like a list of modules that are loaded on our hobbit test page, we can add all those modules to the ‘$DATA’ variable, and finish off this example test.  Within that last look we added, you’d just push that modules name into ‘$DATA’:

[...]
$status = $bbtest . " NOT OK"
}

$DATA .= $module . "\n";
}

So what kindof output does this give us?  If you want to just see what would be run in the system() call at the bottom of the script, just change ‘system’ to ‘print’, the output should be something like the following:

/home/hobbit/client/bin/bb hobbitserver.xipnet.net 'status apollo.modules red Sat May  9 10:35:18 EDT 2009 - modules NOT OK

Module
iptable_filter
ip_tables
rfcomm
l2cap
bluetooth
af_packet
nf_conntrack_netbios_ns
ipt_REJECT
nf_conntrack_ipv4
xt_state
nf_conntrack
ip6t_REJECT
xt_tcpudp
ip6table_filter
ip6_tables
x_tables
ipv6
binfmt_misc
dm_multipath
parport_pc
lp
parport
nvram
evbug
usbcore
ext3
jbd
mbcache
evdev
raid10
raid456
async_xor
async_memcpy
async_tx
xor
raid1
raid0
multipath
linear
md_mod
dm_mirror
dm_snapshot
dm_mod
fuse
loop
8250
serial_core
'

Now the test is red and has the status of “NOT OK” since we tested for any ext file system and I’m running ext3.   To have the data sent to hobbit, simply set the print back to a system call and setup the approprate section within ‘clientlaunch.cfg’, for our example I’d use:

[modules]
ENVFILE $HOBBITCLIENTHOME/etc/hobbitclient.cfg
CMD $HOBBITCLIENTHOME/ext/bb-modules.pl
LOGFILE $HOBBITCLIENTHOME/logs/hobbitclient.log
INTERVAL 5m

I know this is a pretty basic example, but its a place to start – just let the imagination go from here on the millions of tests you could come up with.  Keep in mind I’m sending this data in on the “status” channel in Xymon – there is also a “data” channel which won’t create a column and with the test approprately setup within hobbit to capture its data to rrd’s, you can create graphs that just show up in the “trends” column if alarming is not necessary.