To make this setup work, first you’ll need a virtual IP to point puppet.machine.com at and setup keepalived to manage it. The keepalived setup was fairly straight forward, both machines were on the same network so I just picked an IP and configured keepalived on each machine. In my case the configurations were as follows:

vrrp_instance VI_1 {
        interface bond0
        state MASTER
        virtual_router_id 51
        priority 101
        authentication {
            auth_type PASS
            auth_pass PASSWORD
        }
        virtual_ipaddress {
                192.168.33.61/26 dev bond0
        }
        notify_master /sw/keepalived/scripts/notify_master.sh
        notify_backup /sw/keepalived/scripts/notify_backup.sh
        notify_fault /sw/keepalived/scripts/notify_backup.sh
}

Basically for bonded interface 0, add the virtual interface of 192.168.33.61/26, and this machine has a priority of 101, or master. The three entries at the bottom are what to run if the machine should fail over or fail back. The other machine has a similar setup, except it has the priority of 100, or backup.

After starting up keepalived on both machines, they negotiate and the machine with the higher priority ends up with the IP enabled on bond0 as a secondary address.

Next I needed something to manage fail-over of the puppet “service” should the interface/IP stay up and just the puppet service fail. For this I decided to use HAProxy. It’s configuration is, again, fairly straight forward. You need to setup a front end to listen on the default puppet master port, and a back end that has both your puppet master machines in it. The configuration is as follows:

global
    log         127.0.0.1 local2
    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    maxconn     10000
    user        haproxy
    group       haproxy
    daemon
    stats socket /var/lib/haproxy/stats

defaults
    log                     global
    option                  redispatch
    retries                 3
    timeout http-request    10s
    timeout queue           1m
    timeout connect         10s
    timeout client          1m
    timeout server          1m
    timeout http-keep-alive 10s
    timeout check           10s
    maxconn                 8000

frontend        puppet 192.168.33.61:8140
    mode tcp
    default_backend     puppet0

backend puppet0
    mode        tcp
    option      ssl-hello-chk
    balance     roundrobin
    server      server1 192.168.33.17:8140 check
    server      server2 192.168.33.27:8140 check backup

This configuration and the installation of haproxy is mirrored on the second server.

Now if you recall I had some scripts that were to run should keepalived feel the need to fail to/from the backup server. Because haproxy binds to the address on start up, you can’t have it running on your backup server. To work around the script is run upon keepalived starting that IP on the system, the script starts haproxy.

With the complicated part done, it was just a matter of mirroring my puppet manifests and files between the two machines, and making sure they stay up to date. Ideally you’d have them on some sort of NAS which is just mounted on both machines which would make them being out of date impossible.

You just need to:

1. Copy over ‘puppet_dashboard.rb’ with a modified HOST line to the other host.
2. Modify /etc/sysconfig/puppetmaster.
3. Turn on reporting on the clients.
4. Restart puppetmaster.

Since the puppet_dashboard.rb is just making an HTTP post to the dashboard server, it can be coming from anywhere.

I’ll start with the tripwire.pp file for puppet, in this file you’ll define your tripwire class and associated files and packages:

class tripwire {
        file { "/etc/tripwire/Makefile":
                owner => root,
                group => root,
                mode => 440,
                source => "puppet:///etc/tripwire/Makefile",
                require => Package[tripwire],
        }
        file { "/etc/tripwire/site.key":
                owner => root,
                group => root,
                mode => 440,
                source => "puppet:///etc/tripwire/site.key",
                require => Package[tripwire],
        }
        file { "/etc/tripwire/twcfg.txt":
                owner => root,
                group => root,
                mode => 440,
                source => "puppet:///etc/tripwire/twcfg.txt",
                require => Package[tripwire],
        }
        file { "/etc/tripwire/twpol.txt":
                owner => root,
                group => root,
                mode => 440,
                source => "puppet:///etc/tripwire/twpol.txt",
                require => Package[tripwire],
        }

        file { "/etc/cron.daily/tripwire-check":
                owner => root,
                group => root,
                mode => 755,
                source => "puppet:///etc/cron.daily/tripwire-check",
                require => Package[tripwire],
        }

        file { "/var/lib/tripwire/tripwire.twd":
                owner => root,
                group => root,
                mode => 440,
        }

        package {
                "tripwire":
                        ensure => "installed",
        }

        exec {
                "make -C /etc/tripwire":
                        cwd => "/etc/tripwire",
                        path => ["/bin", "/usr/bin" ],
                        creates => "/var/lib/tripwire/tripwire.twd",
                        timeout => 10000
        }

        exec {
                "tripwire --update-policy -Z low --local-passphrase '' --site-passphrase '' --quiet /etc/tripwire/twpol.txt":
                        cwd => "/etc/tripwire",
                        path => ["/sbin", "/usr/sbin" ],
                        timeout => 10000,
                        refreshonly => "true",
                        subscribe => File["/etc/tripwire/twpol.txt"],

        }
}

You’ll notice there is a Makefile defined, this will make remote management much simpler, it contains some simple house keeping items. I do not use all these within my puppet implementation, however:

all: tw.cfg tw.pol /var/lib/tripwire/tripwire.twd

clean:
        /bin/rm -f /etc/tripwire/local.key /etc/tripwire/tw.cfg /etc/tripwire/tw.pol /var/lib/tripwire/tripwire.twd

init: clean /var/lib/tripwire/tripwire.twd

local.key:
        /usr/sbin/twadmin --generate-keys --local-keyfile local.key --local-passphrase ''

tw.cfg: site.key twcfg.txt
        /usr/sbin/twadmin --create-cfgfile --site-keyfile site.key --site-passphrase '' twcfg.txt

tw.pol: tw.cfg twpol.txt
        if [ -f tw.pol -a -f /var/lib/tripwire/tripwire.twd ]; \
        then /usr/sbin/tripwire --update-policy --local-passphrase '' --quiet --secure-mode low --site-passphrase '' twpol.txt; \
        else /usr/sbin/twadmin --create-polfile --site-passphrase '' twpol.txt; \
        fi

/var/lib/tripwire/tripwire.twd: local.key site.key
        /usr/sbin/tripwire --init --local-passphrase ''

My twcfg.txt is very simple:

# required
DBFILE                 = /var/lib/tripwire/tripwire.twd
LOCALKEYFILE           = /etc/tripwire/local.key
POLFILE                = /etc/tripwire/tw.pol
REPORTFILE             = /var/lib/tripwire/report/$(DATE).twr
SITEKEYFILE            = /etc/tripwire/site.key

# e-mail notification
EMAILREPORTLEVEL       = 3
GLOBALEMAIL            = email@domain.tld
MAILMETHOD             = SENDMAIL
MAILNOVIOLATIONS       = false
MAILPROGRAM            = /usr/sbin/sendmail -oi -t

# other
EDITOR                 = /usr/bin/vim
LATEPROMPTING          = false
LOOSEDIRECTORYCHECKING = false
REPORTLEVEL            = 4
SYSLOGREPORTING        = true
TEMPDIRECTORY          = /var/tmp

Other than including the tripwire class within your site.pp file, thats about it.

Credit: The makefile was created by Steve Coile.