To make this setup work, first you’ll need a virtual IP to point puppet.machine.com at and setup keepalived to manage it. The keepalived setup was fairly straight forward, both machines were on the same network so I just picked an IP and configured keepalived on each machine. In my case the configurations were as follows:

vrrp_instance VI_1 {
        interface bond0
        state MASTER
        virtual_router_id 51
        priority 101
        authentication {
            auth_type PASS
            auth_pass PASSWORD
        }
        virtual_ipaddress {
                192.168.33.61/26 dev bond0
        }
        notify_master /sw/keepalived/scripts/notify_master.sh
        notify_backup /sw/keepalived/scripts/notify_backup.sh
        notify_fault /sw/keepalived/scripts/notify_backup.sh
}

Basically for bonded interface 0, add the virtual interface of 192.168.33.61/26, and this machine has a priority of 101, or master. The three entries at the bottom are what to run if the machine should fail over or fail back. The other machine has a similar setup, except it has the priority of 100, or backup.

After starting up keepalived on both machines, they negotiate and the machine with the higher priority ends up with the IP enabled on bond0 as a secondary address.

Next I needed something to manage fail-over of the puppet “service” should the interface/IP stay up and just the puppet service fail. For this I decided to use HAProxy. It’s configuration is, again, fairly straight forward. You need to setup a front end to listen on the default puppet master port, and a back end that has both your puppet master machines in it. The configuration is as follows:

global
    log         127.0.0.1 local2
    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    maxconn     10000
    user        haproxy
    group       haproxy
    daemon
    stats socket /var/lib/haproxy/stats

defaults
    log                     global
    option                  redispatch
    retries                 3
    timeout http-request    10s
    timeout queue           1m
    timeout connect         10s
    timeout client          1m
    timeout server          1m
    timeout http-keep-alive 10s
    timeout check           10s
    maxconn                 8000

frontend        puppet 192.168.33.61:8140
    mode tcp
    default_backend     puppet0

backend puppet0
    mode        tcp
    option      ssl-hello-chk
    balance     roundrobin
    server      server1 192.168.33.17:8140 check
    server      server2 192.168.33.27:8140 check backup

This configuration and the installation of haproxy is mirrored on the second server.

Now if you recall I had some scripts that were to run should keepalived feel the need to fail to/from the backup server. Because haproxy binds to the address on start up, you can’t have it running on your backup server. To work around the script is run upon keepalived starting that IP on the system, the script starts haproxy.

With the complicated part done, it was just a matter of mirroring my puppet manifests and files between the two machines, and making sure they stay up to date. Ideally you’d have them on some sort of NAS which is just mounted on both machines which would make them being out of date impossible.

Using 1 Core:

[user@server source]$ pbzip2 -p1 -v -m1000 source.tar
Parallel BZIP2 v1.1.1 - by: Jeff Gilchrist [http://compression.ca]
[Apr. 17, 2010]             (uses libbzip2 by Julian Seward)

# CPUs: 1
BWT Block Size: 900 KB
File Block Size: 900 KB
Maximum Memory: 1000 MB
-------------------------------------------
File: 1 of 1
Input Name: source.tar
Output Name: source.tar.bz2

Input Size: 445163520 bytes
Compressing data (no threads)...
Output Size: 80063773 bytes
-------------------------------------------

Wall Clock: 95.100318 seconds

Using 64 Cores:

[user@server source]$ pbzip2 -p64 -v -m1000 source.tar
Parallel BZIP2 v1.1.1 - by: Jeff Gilchrist [http://compression.ca]
[Apr. 17, 2010]             (uses libbzip2 by Julian Seward)

# CPUs: 64
BWT Block Size: 900 KB
File Block Size: 900 KB
Maximum Memory: 1000 MB
-------------------------------------------
File: 1 of 1
Input Name: source.tar
Output Name: source.tar.bz2

Input Size: 445163520 bytes
Compressing data...
Output Size: 80063773 bytes
-------------------------------------------

Wall Clock: 3.795763 seconds

Needless to say I’ve found a new utility for my compression needs.

You just need to:

1. Copy over ‘puppet_dashboard.rb’ with a modified HOST line to the other host.
2. Modify /etc/sysconfig/puppetmaster.
3. Turn on reporting on the clients.
4. Restart puppetmaster.

Since the puppet_dashboard.rb is just making an HTTP post to the dashboard server, it can be coming from anywhere.

My next trick is useful for getting around having your HTTP usage tracked and filtered, this is just taking advantage of the “-D” option within the ‘ssh’ program.  When you invoke the -D option, followed with a port number you not only connect to the remote machine via SSH, but you also create a SOCKS proxy server on the port specified.  Setting up your browser’s proxy settings to point at that port on your local machine will securely forward any requests over to the remote machine, where they will then go out to their final destination.

When you combine corkscrew with ssh -D, you end up with a single, secure connection through the HTTP proxy to your remote machine with a SOCKS proxy through it.  This effectively gives you an easy unfiltered hole through to the outside.

You can also use this trick for getting rsync through a http firewall that might otherwise block it.  To accomplish this you just have setup your RSYNC_CONNECT_PROG environment variable to “/path/to/corkscrew <proxy> <proxy_port> %H 873 <auth_file>”.

Disclaimer: Check with your IT policies before attempting any of the above to make sure its allowed!

./asu64 set SYSTEM_PROD_DATA.SysInfoSerialNum serial_number
./asu64 set SYSTEM_PROD_DATA.SysInfoProdName type_model

Now obviously you’ll be replacing serial_number and type_model with what you want to update the machine too, so don’t just run those commands outright.

Another note of interest was that the machine originally did not have the latest UEFI or IMM firmware on them – and the update would not take – so update everything before attempting this.

For those interested you can also run a ‘show’ on the above information to get the serial and model – I mention this because it does not look like dmidecode works for reading that information any longer.

I’ll start with the tripwire.pp file for puppet, in this file you’ll define your tripwire class and associated files and packages:

class tripwire {
        file { "/etc/tripwire/Makefile":
                owner => root,
                group => root,
                mode => 440,
                source => "puppet:///etc/tripwire/Makefile",
                require => Package[tripwire],
        }
        file { "/etc/tripwire/site.key":
                owner => root,
                group => root,
                mode => 440,
                source => "puppet:///etc/tripwire/site.key",
                require => Package[tripwire],
        }
        file { "/etc/tripwire/twcfg.txt":
                owner => root,
                group => root,
                mode => 440,
                source => "puppet:///etc/tripwire/twcfg.txt",
                require => Package[tripwire],
        }
        file { "/etc/tripwire/twpol.txt":
                owner => root,
                group => root,
                mode => 440,
                source => "puppet:///etc/tripwire/twpol.txt",
                require => Package[tripwire],
        }

        file { "/etc/cron.daily/tripwire-check":
                owner => root,
                group => root,
                mode => 755,
                source => "puppet:///etc/cron.daily/tripwire-check",
                require => Package[tripwire],
        }

        file { "/var/lib/tripwire/tripwire.twd":
                owner => root,
                group => root,
                mode => 440,
        }

        package {
                "tripwire":
                        ensure => "installed",
        }

        exec {
                "make -C /etc/tripwire":
                        cwd => "/etc/tripwire",
                        path => ["/bin", "/usr/bin" ],
                        creates => "/var/lib/tripwire/tripwire.twd",
                        timeout => 10000
        }

        exec {
                "tripwire --update-policy -Z low --local-passphrase '' --site-passphrase '' --quiet /etc/tripwire/twpol.txt":
                        cwd => "/etc/tripwire",
                        path => ["/sbin", "/usr/sbin" ],
                        timeout => 10000,
                        refreshonly => "true",
                        subscribe => File["/etc/tripwire/twpol.txt"],

        }
}

You’ll notice there is a Makefile defined, this will make remote management much simpler, it contains some simple house keeping items. I do not use all these within my puppet implementation, however:

all: tw.cfg tw.pol /var/lib/tripwire/tripwire.twd

clean:
        /bin/rm -f /etc/tripwire/local.key /etc/tripwire/tw.cfg /etc/tripwire/tw.pol /var/lib/tripwire/tripwire.twd

init: clean /var/lib/tripwire/tripwire.twd

local.key:
        /usr/sbin/twadmin --generate-keys --local-keyfile local.key --local-passphrase ''

tw.cfg: site.key twcfg.txt
        /usr/sbin/twadmin --create-cfgfile --site-keyfile site.key --site-passphrase '' twcfg.txt

tw.pol: tw.cfg twpol.txt
        if [ -f tw.pol -a -f /var/lib/tripwire/tripwire.twd ]; \
        then /usr/sbin/tripwire --update-policy --local-passphrase '' --quiet --secure-mode low --site-passphrase '' twpol.txt; \
        else /usr/sbin/twadmin --create-polfile --site-passphrase '' twpol.txt; \
        fi

/var/lib/tripwire/tripwire.twd: local.key site.key
        /usr/sbin/tripwire --init --local-passphrase ''

My twcfg.txt is very simple:

# required
DBFILE                 = /var/lib/tripwire/tripwire.twd
LOCALKEYFILE           = /etc/tripwire/local.key
POLFILE                = /etc/tripwire/tw.pol
REPORTFILE             = /var/lib/tripwire/report/$(DATE).twr
SITEKEYFILE            = /etc/tripwire/site.key

# e-mail notification
EMAILREPORTLEVEL       = 3
GLOBALEMAIL            = email@domain.tld
MAILMETHOD             = SENDMAIL
MAILNOVIOLATIONS       = false
MAILPROGRAM            = /usr/sbin/sendmail -oi -t

# other
EDITOR                 = /usr/bin/vim
LATEPROMPTING          = false
LOOSEDIRECTORYCHECKING = false
REPORTLEVEL            = 4
SYSLOGREPORTING        = true
TEMPDIRECTORY          = /var/tmp

Other than including the tripwire class within your site.pp file, thats about it.

Credit: The makefile was created by Steve Coile.

• Modified JavaScript so it would load & function in IE
• Changed “class” to “clas” within JS, Perl, and the database. IE didn’t like using the word “class” as a variable name, changed else where for consistency.
• Stopped opening of “right-click” menu when pressed on an interface folder
• Fixed DB Schema to include basic information for default OIDs and Interfaces.
• Modified Apache configuration so SSIs would work correctly in Apache 2, changed the SSIs to only execute on .shtml files
• Renamed index.html to index.shtml

There is a database change, so you must run the sql file located in the upgrade folder!

You can download it here.

The SourceForge project page is located here.
The project home page is located here.

IBM BladeCenters

> Get the key from the BladeCenter

1. Login to the BladeCenter
2. Expand “MM Control”
3. Click “Network Protocols”
4. Click “Secure Shell (SSH) Server”
5. Under the SSL for “Web Server click Generate” a “Generate a New Server Key and Certificate Signing Request (CSR)”
6. You’ll need to fill out the requested information down to SurName, then click “Generate CSR”
7. Save the key to your machine, prepend machine. (bladecenter.) to the beginning of the file name to denote where it came from

> SCP the file to root@server:/path/to/your/CA/requests/
> To convert the DER encoded CSR to a PEM encoded CSR, within the above folder, run:

cd /path/to/your/CA/requests
openssl req -inform DER -in bladecenter.csr_server.der -out bladecenter.csr.pem -outform PEM

> To sign the key, run:

cd /path/to/your/CA
openssl ca -config openssl.cnf -policy policy_anything -out certs/bladecenter.crt -infiles requests/bladecenter.csr.pem

> Convert the created .crt file to a .der file, run:

cd /path/to/your/CA/certs
openssl x509 -in bladecenter.crt -inform PEM -out bladecenter.crt.der -outform DER

> SCP the cert file (/path/to/your/CA/certs/bladecenter.crt.der) back to your system
> Import the cert into the AMM

1. Click “Import a Signed Certificate to the Server”
2. Browse for the file you just copied back from server (bladecenter.crt.der)
3. Click “Import Server Certificate”

IBM RSA II

> Get the key from the RSA II Controller

1. Login to the RSA II Controller
2. Click “Security”
3. Under the SSL for “Web Server click Generate” a “Generate a New Server Key and Certificate Signing Request (CSR)”
4. You’ll need to fill out the requested information down to SurName, then click “Generate CSR”

> Save the key to your machine, prepend machine. (rsa.) to the beginning of the file name to denote where it came from
> SCP the file to root@server:/path/to/your/CA/requests/
> To convert the DER encoded CSR to a PEM encoded CSR, within the above folder, run:

cd /path/to/your/CA/requests
openssl req -inform DER -in rsa.csr_server.der -out rsa.csr.pem -outform PEM

> To sign the key, run:

cd /path/to/your/CA
openssl ca -config openssl.cnf -policy policy_anything -out certs/rsa.crt -infiles requests/rsa.csr.pem

> Convert the created .crt file to a .der file, run:

cd /path/to/your/CA/certs
openssl x509 -in rsa.crt -inform PEM -out rsa.crt.der -outform DER

> SCP the cert file (/path/to/your/CA/rsa.crt.der) back to your system
> Import the cert into the RSA II Controller

1. Click “Import a Signed Certificate to the Server”
2. Browse for the file you just copied back from server (rsa.crt.der)
3. Click “Import Server Certificate”

HP iLO

> Get the Signing Request from the iLO

1. Login to the iLO
2. Hover over “Administration”
3. Select “Certificate Administration”
4. Click “Create Certificate Request”
5. Copy and paste the request from the box and into server:/path/to/your/CA/requests/ilo.csr.pem

> To sign the key, run:

cd /path/to/your/CA
openssl ca -config openssl.cnf -policy policy_anything -out certs/ilo.crt -infiles requests/ilo.csr.pem

> Import the cert into the iLO

1. Click “Import Certificate”
2. Enter the contents of /path/to/your/CA/certs/ilo.crt into the provided box
3. Click “Next”

Sun eLOM

> Get the Signing Request from the eLOM

1. Login to the eLOM
2. Click “Configuration”
3. Click “System Management Access”
4. Click “SSL Certificate”
5. Select “Certificate” and click “Select”

> Generate the request:

cd /path/to/your/CA
openssl req -config openssl.cnf -new -nodes -keyout private/elom.key -out requests/elom.csr -days 365

You’ll be asked specific questions about the servers location, most of the defaults have been set in openssl.cnf, however you’ll need to make sure the domain name (CN) is correct.

> Sign the key:

cd /path/to/your/CA
openssl ca -config openssl.cnf -policy policy_anything -out certs/elom.crt -infiles requests/elom.csr

> Import the Key on the eLOM

1. Upload the generated Certificate first (server:/path/to/your/CA/certs/elom.crt)
2. Upload the generated Key Second (server:/path/to/your/CA/private/elom.key)

Dell DRAC

> Get the Signing Request from the DRAC

1. Login to the DRAC
2. Click the “Configuration” tab
3. Click the “Security” tab
4. Select “Generate a new Certificate Signing Request (CSR)”
5. Click “Next”
6. Complete the listed fields making sure that the “Common Name” (CN) is the full host name of the console
7. Click “Generate”, wait for it to generate the request. It will pop up with a download for “csr.txt” when it is complete. Save the file.
8. Rename csr.txt to include the hostname of the box, EX: drac.csr.txt
9. SCP the file to root@server:/path/to/your/CA/requests

> To sign the key, run:

cd /path/to/your/CA
openssl ca -config openssl.cnf -policy policy_anything -out certs/drac.crt -infiles requests/drac.csr.txt

> Edit the crt file to remove everything above the “—–BEGIN CERTIFICATE—–” line
> Save the File with DOS-style Line-Ending CR-LF (:set ff=dos in vi)
> Import the cert into the DRAC

1. Return to “Certificate Management”
2. Select “Upload DRAC 4 server certificate”
3. Click “Next”
4. Select the certificate you just signed
5. Click “Upload”

After a year of this mess I decided to go a different route – a dynamic web application that would update it self based on data pulled from the blade centers via SNMP.  Glovebox was born.  Well, actually it was called “the dashboard” in the beginning – unfortunately for me my co-workers have no issue pointing out when things aren’t named quite right, a sarcastic “well it ain’t a dashboard… how about Glovebox!” decided the name fate of it.   The product took almost a year to get to its current self, but manages to keep track of all our blades, IBM RSA II consoles and our Xen virtual machines without the input from anyone on staff.

For the IBM blades and RSA II cards it uses a set of OIDs I gathered from sifting through their respective snmp output.  In the case of the blades, those OIDs are:

| 1.3.6.1.4.1.2.3.51.2.2.21.1.1.3    | bladecenters | serial          |
| 1.3.6.1.4.1.2.3.51.2.2.8.1.1       | bladecenters | error           |
| 1.3.6.1.4.1.2.3.51.2.2.21.1.1.2    | bladecenters | family          |
| 1.3.6.1.4.1.2.3.51.2.2.21.1.1.1    | bladecenters | model           |
| 1.3.6.1.4.1.2.3.51.2.22.3.1.1.1.15 | switches     | error           |
| 1.3.6.1.4.1.2.3.51.2.22.3.1.7.1.6  | switches     | address         |
| 1.3.6.1.4.1.2.3.51.2.2.21.5.1.1.6  | servers      | bios            |
| 1.3.6.1.4.1.2.3.51.2.2.21.4.1.1.6  | servers      | serial          |
| 1.3.6.1.4.1.2.3.51.2.2.21.4.1.1.7  | servers      | model           |
| 1.3.6.1.4.1.2.3.51.2.22.1.5.1.1.6  | servers      | hostname        |
| 1.3.6.1.4.1.2.3.51.2.2.21.5.3.1.6  | servers      | bsmp            |
| 1.3.6.1.4.1.2.3.51.2.2.8.2.1.1.7   | servers      | error           |
| 1.3.6.1.4.1.2.3.51.2.2.8.2.1.1.4   | servers      | power           |
| 1.3.6.1.4.1.2.3.51.2.2.21.4.1.1.20 | servers      | storage         |
| 1.3.6.1.4.1.2.3.51.2.2.21.4.1.1.12 | servers      | family          |
| 1.3.6.1.4.1.2.3.51.2.22.3.1.7.1.5  | switches     | serial          |
| 1.3.6.1.4.1.2.3.51.2.3.4.2.1.2     | bladecenters | eventlog        |
| 1.3.6.1.4.1.2.3.51.2.4.5.1         | bladecenters | name            |
| 1.3.6.1.4.1.2.3.51.2.2.10.5.1.2    | bladecenters | watts           |
| 1.3.6.1.4.1.2.3.51.2.2.10.5.1.3    | bladecenters | btus            |
| 1.3.6.1.4.1.2.3.51.2.4.9.1.1.3     | bladecenters | hostname        |
| 1.3.6.1.4.1.2.3.51.2.2.8.1.2       | bladecenters | infoled         |
| 1.3.6.1.4.1.2.3.51.2.2.8.1.3       | bladecenters | templed         |
| 1.3.6.1.4.1.2.3.51.2.2.8.1.4       | bladecenters | identled        |
| 1.3.6.1.4.1.2.3.51.2.22.4.25       | bladecenters | installedblades |
| 1.3.6.1.4.1.2.3.51.2.2.21.3.1.1.3  | mm           | firmware        |
| 1.3.6.1.4.1.2.3.51.2.22.4.30       | bladecenters | installedmms    |
| 1.3.6.1.4.1.2.3.51.2.2.21.2.1.1.9  | mm           | serial          |
| 1.3.6.1.4.1.2.3.51.2.2.21.3.1.1.6  | mm           | firmware_date   |
| 1.3.6.1.4.1.2.3.51.2.2.21.3.1.1.2  | mm           | name            |
| 1.3.6.1.4.1.2.3.51.2.22.5.1.1.3    | mm           | address         |
| 1.3.6.1.4.1.2.3.51.2.22.5.1.1.5    | mm           | error           |
| 1.3.6.1.4.1.2.3.51.2.22.5.1.1.4    | mm           | primary         |

From the list you can see I monitor not just the blades but also the management modules, the chassis, and the blade switches.  The snmp output on the IBM Advanced Management Modules is quite complete and they provide good MIBs to be able to decipher the information.

I have a similar list for the RSA II cards which was added after a re-write of the initial version of Glovebox mid last year, the re-write was to make the software be able to use “modules” for each device, this was so adding future additions was easy.   The Xen virtual machines are added via libvirt and its associated Perl module, I have setup keys between the server that this application runs on and each Domain-0, allowing it connection and figure out whats running on there.   Since it based off modules you very well could add anything you wanted – it even has a shared development libary that allows for easy additions without re-inventing the wheel.

So whats it look like?  Well it looks like any application written with ExtJS as the front end – quite good.  I’m not a designer, I can’t do graphics, but with the help of ExtJS it came out quite nicely.  How about a peak:

Sorry for blocking out some of the data, but some of the info needs to stay in house. The back end is entirely Perl and all data is kept in a MySQL database.  It ended up being a rather larger application in the end, but really makes life quite a bit easier.