• Modified JavaScript so it would load & function in IE
• Changed “class” to “clas” within JS, Perl, and the database. IE didn’t like using the word “class” as a variable name, changed else where for consistency.
• Stopped opening of “right-click” menu when pressed on an interface folder
• Fixed DB Schema to include basic information for default OIDs and Interfaces.
• Modified Apache configuration so SSIs would work correctly in Apache 2, changed the SSIs to only execute on .shtml files
• Renamed index.html to index.shtml

There is a database change, so you must run the sql file located in the upgrade folder!

You can download it here.

The SourceForge project page is located here.
The project home page is located here.

Since I’ve had to write quite a few of these I thought I’d share a simple recipe for getting started with these scripts.   Most all our scripts are written in Perl, its just what I’m used too and what most people in our company is fluent in.   Our older scripts (from when we actually ran Big Brother) are almost all shell scripts and I have written at least one in Python since that particular service’s library was in python.   For this example I’m going to use Perl default template:

#!/usr/bin/perl -w
#############################################################################
# $Id: $
#############################################################################
use strict;

## BB and related test constants
#############################################################################

use constant GREEN => 'green';
use constant YELLOW => 'yellow';
use constant RED => 'red';

## BB Global variables
#############################################################################

my $prev_run_log = "$ENV{BBTMP}/$ENV{MACHINE}.lighttpd.data";
my $bbtest = 'testname';
my $color = GREEN;
my $status = $bbtest . " OK";

## Main Program
#############################################################################
{

my $DATA = "";

## TEST STUFF HERE

## Send to Hobbit
#############################################################################
my $report_date = `/bin/date`;
chomp($report_date);

system("$ENV{BB} $ENV{BBDISP} 'status $ENV{MACHINE}.$bbtest $color $report_date - $status\n\n$DATA'\n");

}

This script by itself will send a test called ‘testname’, which is green and has the status of ‘OK’.  Since we have not defined anything to test it won’t contain any data with it.  To run this script you need to be within Xymon’s “shell”, which is basically sh, with some enviroment variables set.  To get there, go to ’bin’ directory within your hobbit client installation, in my case this is: “/home/hobbit/client/bin”, you’ll need to run ‘bbcmd’, this will drop you to a prompt which looks something like this:

[rar@apollo bin]$ ./bbcmd
2009-05-09 10:10:35 Using default environment file /home/hobbit/client/etc/hobbitclient.cfg
sh-3.2$

Lets change the test name to something else and gather some data from the system we are running on.   To change the test name to say ‘modules’, just change the $bbtest to ‘modules’.   How about testing what kernel modules are loaded – or basically just parse the output of ‘lsmod’.   To do this we’d need to capture the output of the command, this is easy enough in perl, something like this would work just fine:

my $lsmod = `/sbin/lsmod`;

With the output of ’lsmod’ captured we can do whatever we want with it.  Lets say we just want the modules, and no other info about them in an array:

my $lsmod = `/sbin/lsmod`;
my @modules = split(/\n/, $lsmod);
my @module_names;

foreach my $module (@modules){
my ($name, $junk) = split(/\s.+/, $module);
push @module_names, $name;
}

Now that we have our array created that we can easily loop through for tests, lets run a test on them to see if we have loaded any modules for ‘ext’ file systems:

foreach my $module (@module_names) {
if ($module =~ "ext"){
$color = RED;
$status = $bbtest . " NOT OK"
}
}

OK,  since we’d like a list of modules that are loaded on our hobbit test page, we can add all those modules to the ‘$DATA’ variable, and finish off this example test.  Within that last look we added, you’d just push that modules name into ‘$DATA’:

[...]
$status = $bbtest . " NOT OK"
}

$DATA .= $module . "\n";
}

So what kindof output does this give us?  If you want to just see what would be run in the system() call at the bottom of the script, just change ‘system’ to ‘print’, the output should be something like the following:

/home/hobbit/client/bin/bb hobbitserver.xipnet.net 'status apollo.modules red Sat May  9 10:35:18 EDT 2009 - modules NOT OK

Module
iptable_filter
ip_tables
rfcomm
l2cap
bluetooth
af_packet
nf_conntrack_netbios_ns
ipt_REJECT
nf_conntrack_ipv4
xt_state
nf_conntrack
ip6t_REJECT
xt_tcpudp
ip6table_filter
ip6_tables
x_tables
ipv6
binfmt_misc
dm_multipath
parport_pc
lp
parport
nvram
evbug
usbcore
ext3
jbd
mbcache
evdev
raid10
raid456
async_xor
async_memcpy
async_tx
xor
raid1
raid0
multipath
linear
md_mod
dm_mirror
dm_snapshot
dm_mod
fuse
loop
8250
serial_core
'

Now the test is red and has the status of “NOT OK” since we tested for any ext file system and I’m running ext3.   To have the data sent to hobbit, simply set the print back to a system call and setup the approprate section within ‘clientlaunch.cfg’, for our example I’d use:

[modules]
ENVFILE $HOBBITCLIENTHOME/etc/hobbitclient.cfg
CMD $HOBBITCLIENTHOME/ext/bb-modules.pl
LOGFILE $HOBBITCLIENTHOME/logs/hobbitclient.log
INTERVAL 5m

I know this is a pretty basic example, but its a place to start – just let the imagination go from here on the millions of tests you could come up with.  Keep in mind I’m sending this data in on the “status” channel in Xymon – there is also a “data” channel which won’t create a column and with the test approprately setup within hobbit to capture its data to rrd’s, you can create graphs that just show up in the “trends” column if alarming is not necessary.

After a year of this mess I decided to go a different route – a dynamic web application that would update it self based on data pulled from the blade centers via SNMP.  Glovebox was born.  Well, actually it was called “the dashboard” in the beginning – unfortunately for me my co-workers have no issue pointing out when things aren’t named quite right, a sarcastic “well it ain’t a dashboard… how about Glovebox!” decided the name fate of it.   The product took almost a year to get to its current self, but manages to keep track of all our blades, IBM RSA II consoles and our Xen virtual machines without the input from anyone on staff.

For the IBM blades and RSA II cards it uses a set of OIDs I gathered from sifting through their respective snmp output.  In the case of the blades, those OIDs are:

| 1.3.6.1.4.1.2.3.51.2.2.21.1.1.3    | bladecenters | serial          |
| 1.3.6.1.4.1.2.3.51.2.2.8.1.1       | bladecenters | error           |
| 1.3.6.1.4.1.2.3.51.2.2.21.1.1.2    | bladecenters | family          |
| 1.3.6.1.4.1.2.3.51.2.2.21.1.1.1    | bladecenters | model           |
| 1.3.6.1.4.1.2.3.51.2.22.3.1.1.1.15 | switches     | error           |
| 1.3.6.1.4.1.2.3.51.2.22.3.1.7.1.6  | switches     | address         |
| 1.3.6.1.4.1.2.3.51.2.2.21.5.1.1.6  | servers      | bios            |
| 1.3.6.1.4.1.2.3.51.2.2.21.4.1.1.6  | servers      | serial          |
| 1.3.6.1.4.1.2.3.51.2.2.21.4.1.1.7  | servers      | model           |
| 1.3.6.1.4.1.2.3.51.2.22.1.5.1.1.6  | servers      | hostname        |
| 1.3.6.1.4.1.2.3.51.2.2.21.5.3.1.6  | servers      | bsmp            |
| 1.3.6.1.4.1.2.3.51.2.2.8.2.1.1.7   | servers      | error           |
| 1.3.6.1.4.1.2.3.51.2.2.8.2.1.1.4   | servers      | power           |
| 1.3.6.1.4.1.2.3.51.2.2.21.4.1.1.20 | servers      | storage         |
| 1.3.6.1.4.1.2.3.51.2.2.21.4.1.1.12 | servers      | family          |
| 1.3.6.1.4.1.2.3.51.2.22.3.1.7.1.5  | switches     | serial          |
| 1.3.6.1.4.1.2.3.51.2.3.4.2.1.2     | bladecenters | eventlog        |
| 1.3.6.1.4.1.2.3.51.2.4.5.1         | bladecenters | name            |
| 1.3.6.1.4.1.2.3.51.2.2.10.5.1.2    | bladecenters | watts           |
| 1.3.6.1.4.1.2.3.51.2.2.10.5.1.3    | bladecenters | btus            |
| 1.3.6.1.4.1.2.3.51.2.4.9.1.1.3     | bladecenters | hostname        |
| 1.3.6.1.4.1.2.3.51.2.2.8.1.2       | bladecenters | infoled         |
| 1.3.6.1.4.1.2.3.51.2.2.8.1.3       | bladecenters | templed         |
| 1.3.6.1.4.1.2.3.51.2.2.8.1.4       | bladecenters | identled        |
| 1.3.6.1.4.1.2.3.51.2.22.4.25       | bladecenters | installedblades |
| 1.3.6.1.4.1.2.3.51.2.2.21.3.1.1.3  | mm           | firmware        |
| 1.3.6.1.4.1.2.3.51.2.22.4.30       | bladecenters | installedmms    |
| 1.3.6.1.4.1.2.3.51.2.2.21.2.1.1.9  | mm           | serial          |
| 1.3.6.1.4.1.2.3.51.2.2.21.3.1.1.6  | mm           | firmware_date   |
| 1.3.6.1.4.1.2.3.51.2.2.21.3.1.1.2  | mm           | name            |
| 1.3.6.1.4.1.2.3.51.2.22.5.1.1.3    | mm           | address         |
| 1.3.6.1.4.1.2.3.51.2.22.5.1.1.5    | mm           | error           |
| 1.3.6.1.4.1.2.3.51.2.22.5.1.1.4    | mm           | primary         |

From the list you can see I monitor not just the blades but also the management modules, the chassis, and the blade switches.  The snmp output on the IBM Advanced Management Modules is quite complete and they provide good MIBs to be able to decipher the information.

I have a similar list for the RSA II cards which was added after a re-write of the initial version of Glovebox mid last year, the re-write was to make the software be able to use “modules” for each device, this was so adding future additions was easy.   The Xen virtual machines are added via libvirt and its associated Perl module, I have setup keys between the server that this application runs on and each Domain-0, allowing it connection and figure out whats running on there.   Since it based off modules you very well could add anything you wanted – it even has a shared development libary that allows for easy additions without re-inventing the wheel.

So whats it look like?  Well it looks like any application written with ExtJS as the front end – quite good.  I’m not a designer, I can’t do graphics, but with the help of ExtJS it came out quite nicely.  How about a peak:

Sorry for blocking out some of the data, but some of the info needs to stay in house. The back end is entirely Perl and all data is kept in a MySQL database.  It ended up being a rather larger application in the end, but really makes life quite a bit easier.

The Graph definition is as follows:

[lighttpd]
TITLE lighttpd Requests/Second
YAXIS # reqs/sec
DEF:RPS=lighttpd.rrd:reqpersec:AVERAGE
LINE2:RPS#0000CC:reqs/sec
COMMENT:
GPRINT:RPS:LAST:Requests per Second   : %5.1lf (cur)
GPRINT:RPS:MAX: : %5.1lf (max)
GPRINT:RPS:MIN: : %5.1lf (min)
GPRINT:RPS:AVERAGE: : %5.1lf (avg)

The Xymon scripts source code is:

#!/usr/bin/perl -w
#############################################################################
# $Id: mi-lighttpd.pl 21 2009-05-07 19:22:28Z rar $
#############################################################################

use strict;
use File::Slurp;
use LWP::Simple;
use Data::Dumper;

my $port = 80;
my $bbtest = 'lighttpd';

#############################################################################
## BB and related test constants
#############################################################################

use constant CLEAR  => 'clear';
use constant GREEN =>  'green';
use constant YELLOW => 'yellow';
use constant RED    => 'red';
use constant PURPLE => 'purple';

{

#############################################################################
## Setup Variables
#############################################################################
	my $DATA    = "";
	my $color   = GREEN;
	my $status  = $bbtest . " OK";
	my $previous_run;
	my $current_run;
	my $restarted   = 0;
	my @delta_tests = ( 'total_kbytes', 'total_accesses', 'uptime' );
	my @counter_tests   = ( 'total_accesses', 'total_kbytes', 'uptime', 'reqpersec' );
	my @write_log;
	my $backup_log    = "$ENV{BBTMP}/$ENV{MACHINE}.lighttpd.data";
	my %lighttpd_states = (
		'_' => 'awaiting_conn',
		'.' => 'connect',
		'r' => 'reading_req',
		'R' => 'reading_req_post',
		's' => 'sending_reply',
		'S' => 'sending_reply_end',
		'q' => 'request_start',
		'Q' => 'request_end',
		'W' => 'write',
		'h' => 'handle_request',
		'C' => 'close',
		'E' => 'hard_error',
	);
	my %error_states = (
		'reqpersec' => '100',
		'sending_reply' => '100'
	);

	my %warn_states = (
		'reqpersec' => '75',
		'sending_reply' => '75'
	);

#############################################################################
## Gather previous run data & current data
#############################################################################

	# Get Last Status
	if ( -e $backup_log ) {

		#Split Apart Data
		my @data = read_file($backup_log);
		if (@data) {
			foreach my $line (@data) {
				chomp($line);
				my ( $key, $value ) = split( /=/, $line, 2 );
				$previous_run->{$key} = $value;
			}
		}
		else {
			$previous_run = 0;
		}
	}

	my $status_data = get "http://localhost:$port/server-status?auto";

#############################################################################
## Munge the data up to get what we want
#############################################################################

	if ( defined($status_data) ) {

		my @data = split( /n/, $status_data );
		foreach my $line (@data) {
			chomp($line);
			my ( $key, $value ) = split( /: /, $line );
			$key =~ s/ /_/g;
			$key = lc($key);
			$current_run->{$key} = $value;
		}

		# Create array of new values to save for our next run
		foreach ( keys %{$current_run} ) {
			push @write_log, "$_=$current_run->{$_}n";
		}

		# Calculate Incremental Data
		if ($previous_run) {
			# We need to check if we restarted here to avoid negitives
			if ($current_run->{uptime} > $previous_run->{uptime} ){
				foreach (@delta_tests) {
						$current_run->{$_} = $current_run->{$_} - $previous_run->{$_};
				}
			}
			else {
				$status = $bbtest . " Restarted";
				$color = YELLOW;
				$restarted = 1;
			}
		}

		# Break out scoreboard data
		foreach ( keys %lighttpd_states ) {
			$current_run->{ $lighttpd_states{$_} } = 0;
		}
		foreach ( split( //, $current_run->{scoreboard} ) ) {
			$current_run->{ $lighttpd_states{$_} }++;
		}

		if ( !$restarted ) {
			# Get Req/Sec since last check
			$current_run->{reqpersec} = sprintf "%.2F", ( $current_run->{total_accesses} / $current_run->{uptime} ) if $current_run->{uptime};

			## save our counter tests for the next run incase we
			## lighttpd gets restarted we have some non-zero data to send in
			foreach (@counter_tests) {
				push @write_log, "cur_$_=$current_run->{$_}n";
			}    

			# Check for warning states
			foreach ( keys %warn_states ) {
				if ( $current_run->{$_} > $warn_states{$_} ) {
					$color  = YELLOW;
					$status = $bbtest . " $_ > $warn_states{$_}";
				}
			}

			# Check for error states
			foreach ( keys %error_states ) {
				if ( $current_run->{$_} > $error_states{$_} ) {
					$color  = RED;
					$status = $bbtest . " $_ > $error_states{$_}";
				}
			}
		}
		else {
			## So we restarted, and we don't want to send nothing
			## and we don't want to send negitives on calculated tests
			## since we saved them last run for this purpose, we'll send in
			## our last run stats to keep from having huge drops in the
			## graphs.
			foreach (@counter_tests) {
				$current_run->{$_} = sprintf "%.2F", $previous_run->{ "cur_".$_ };
			}
		}

		# Delete Some Unwanted Data
		delete $current_run->{scoreboard};
		delete $current_run->{uptime};

		# Create Hobbit Data
		foreach ( sort keys %{$current_run} ) {
			$DATA .= "$_:$current_run->{$_}n";
		}

		# Actually Write last run file
		write_file( $backup_log, @write_log );

	}
	else {
		$status = $bbtest . " unable to get server-status";
		$color  = CLEAR;
	}

#############################################################################
## Give data to hobbit
#############################################################################

	# hobbit formatted output
	my $report_date = `/bin/date`;
	chomp($report_date);

	## DEV DEBUG
	#print "$ENV{BB} $ENV{BBDISP} 'status apollo,xipnet,net.$bbtest $color $report_date - $statusnn$DATA'n";
	system("$ENV{BB} $ENV{BBDISP} 'status apollo,xipnet,net.$bbtest $color $report_date - $statusnn$DATA'n");
}