Ball Dawg! » Server Management

Parallel BZIP2: pbzip2

Andrew Rankin — Wed, 07 Jul 2010 18:53:22 +0000

I find myself compressing files for archival purposes constantly – today I was sitting, waiting on one such compression on a SMP box and thought it seems silly that bzip2 does not use more than one CPU. After a quick flip through the man page for bzip2 I found no way to force it to use more than one core. A quick web search yielded pbzip2 (http://compression.ca/pbzip2/) – another project that does indeed allow you to use more than one CPU for compress and decompression of bzip2 files. A quick test showed a huge reduction in compression time:

Using 1 Core:

[user@server source]$ pbzip2 -p1 -v -m1000 source.tar
Parallel BZIP2 v1.1.1 - by: Jeff Gilchrist [http://compression.ca]
[Apr. 17, 2010]             (uses libbzip2 by Julian Seward)

# CPUs: 1
BWT Block Size: 900 KB
File Block Size: 900 KB
Maximum Memory: 1000 MB
-------------------------------------------
File: 1 of 1
Input Name: source.tar
Output Name: source.tar.bz2

Input Size: 445163520 bytes
Compressing data (no threads)...
Output Size: 80063773 bytes
-------------------------------------------

Wall Clock: 95.100318 seconds

Using 64 Cores:

[user@server source]$ pbzip2 -p64 -v -m1000 source.tar
Parallel BZIP2 v1.1.1 - by: Jeff Gilchrist [http://compression.ca]
[Apr. 17, 2010]             (uses libbzip2 by Julian Seward)

# CPUs: 64
BWT Block Size: 900 KB
File Block Size: 900 KB
Maximum Memory: 1000 MB
-------------------------------------------
File: 1 of 1
Input Name: source.tar
Output Name: source.tar.bz2

Input Size: 445163520 bytes
Compressing data...
Output Size: 80063773 bytes
-------------------------------------------

Wall Clock: 3.795763 seconds

Needless to say I’ve found a new utility for my compression needs.

Glovebox v0.2.1 Released

Andrew Rankin — Sat, 13 Jun 2009 23:06:02 +0000

I’ve released Glovebox Version 0.2.1 on Sourceforge. It has a few bug fixes as well as the correct database schema included. Other changes include:

Modified JavaScript so it would load & function in IE
Changed “class” to “clas” within JS, Perl, and the database. IE didn’t like using the word “class” as a variable name, changed else where for consistency.
Stopped opening of “right-click” menu when pressed on an interface folder
Fixed DB Schema to include basic information for default OIDs and Interfaces.
Modified Apache configuration so SSIs would work correctly in Apache 2, changed the SSIs to only execute on .shtml files
Renamed index.html to index.shtml

There is a database change, so you must run the sql file located in the upgrade folder!

You can download it here.

Glovebox: My Solution to Managing Servers

Andrew Rankin — Sat, 09 May 2009 01:39:05 +0000

When I started in the “Technology” department at my current employer, I found myself apart of a team that was tasked with taking care of hundreds of IBM blade servers, and tens of other IBM system x servers. For the most part we could keep up with our servers by where they were in our monitoring software, but if we needed to know exactly where they were in either a blade center, by remote console name, or what Domain-0 they lived on for our Xen based virtual machines – we had to relate back to a usually out of date spreadsheet that showed where to go.

After a year of this mess I decided to go a different route – a dynamic web application that would update it self based on data pulled from the blade centers via SNMP. Glovebox was born. Well, actually it was called “the dashboard” in the beginning – unfortunately for me my co-workers have no issue pointing out when things aren’t named quite right, a sarcastic “well it ain’t a dashboard… how about Glovebox!” decided the name fate of it. The product took almost a year to get to its current self, but manages to keep track of all our blades, IBM RSA II consoles and our Xen virtual machines without the input from anyone on staff.

For the IBM blades and RSA II cards it uses a set of OIDs I gathered from sifting through their respective snmp output. In the case of the blades, those OIDs are:

| 1.3.6.1.4.1.2.3.51.2.2.21.1.1.3    | bladecenters | serial          |
| 1.3.6.1.4.1.2.3.51.2.2.8.1.1       | bladecenters | error           |
| 1.3.6.1.4.1.2.3.51.2.2.21.1.1.2    | bladecenters | family          |
| 1.3.6.1.4.1.2.3.51.2.2.21.1.1.1    | bladecenters | model           |
| 1.3.6.1.4.1.2.3.51.2.22.3.1.1.1.15 | switches     | error           |
| 1.3.6.1.4.1.2.3.51.2.22.3.1.7.1.6  | switches     | address         |
| 1.3.6.1.4.1.2.3.51.2.2.21.5.1.1.6  | servers      | bios            |
| 1.3.6.1.4.1.2.3.51.2.2.21.4.1.1.6  | servers      | serial          |
| 1.3.6.1.4.1.2.3.51.2.2.21.4.1.1.7  | servers      | model           |
| 1.3.6.1.4.1.2.3.51.2.22.1.5.1.1.6  | servers      | hostname        |
| 1.3.6.1.4.1.2.3.51.2.2.21.5.3.1.6  | servers      | bsmp            |
| 1.3.6.1.4.1.2.3.51.2.2.8.2.1.1.7   | servers      | error           |
| 1.3.6.1.4.1.2.3.51.2.2.8.2.1.1.4   | servers      | power           |
| 1.3.6.1.4.1.2.3.51.2.2.21.4.1.1.20 | servers      | storage         |
| 1.3.6.1.4.1.2.3.51.2.2.21.4.1.1.12 | servers      | family          |
| 1.3.6.1.4.1.2.3.51.2.22.3.1.7.1.5  | switches     | serial          |
| 1.3.6.1.4.1.2.3.51.2.3.4.2.1.2     | bladecenters | eventlog        |
| 1.3.6.1.4.1.2.3.51.2.4.5.1         | bladecenters | name            |
| 1.3.6.1.4.1.2.3.51.2.2.10.5.1.2    | bladecenters | watts           |
| 1.3.6.1.4.1.2.3.51.2.2.10.5.1.3    | bladecenters | btus            |
| 1.3.6.1.4.1.2.3.51.2.4.9.1.1.3     | bladecenters | hostname        |
| 1.3.6.1.4.1.2.3.51.2.2.8.1.2       | bladecenters | infoled         |
| 1.3.6.1.4.1.2.3.51.2.2.8.1.3       | bladecenters | templed         |
| 1.3.6.1.4.1.2.3.51.2.2.8.1.4       | bladecenters | identled        |
| 1.3.6.1.4.1.2.3.51.2.22.4.25       | bladecenters | installedblades |
| 1.3.6.1.4.1.2.3.51.2.2.21.3.1.1.3  | mm           | firmware        |
| 1.3.6.1.4.1.2.3.51.2.22.4.30       | bladecenters | installedmms    |
| 1.3.6.1.4.1.2.3.51.2.2.21.2.1.1.9  | mm           | serial          |
| 1.3.6.1.4.1.2.3.51.2.2.21.3.1.1.6  | mm           | firmware_date   |
| 1.3.6.1.4.1.2.3.51.2.2.21.3.1.1.2  | mm           | name            |
| 1.3.6.1.4.1.2.3.51.2.22.5.1.1.3    | mm           | address         |
| 1.3.6.1.4.1.2.3.51.2.22.5.1.1.5    | mm           | error           |
| 1.3.6.1.4.1.2.3.51.2.22.5.1.1.4    | mm           | primary         |

From the list you can see I monitor not just the blades but also the management modules, the chassis, and the blade switches. The snmp output on the IBM Advanced Management Modules is quite complete and they provide good MIBs to be able to decipher the information.

I have a similar list for the RSA II cards which was added after a re-write of the initial version of Glovebox mid last year, the re-write was to make the software be able to use “modules” for each device, this was so adding future additions was easy. The Xen virtual machines are added via libvirt and its associated Perl module, I have setup keys between the server that this application runs on and each Domain-0, allowing it connection and figure out whats running on there. Since it based off modules you very well could add anything you wanted – it even has a shared development libary that allows for easy additions without re-inventing the wheel.

So whats it look like? Well it looks like any application written with ExtJS as the front end – quite good. I’m not a designer, I can’t do graphics, but with the help of ExtJS it came out quite nicely. How about a peak:

Sorry for blocking out some of the data, but some of the info needs to stay in house. The back end is entirely Perl and all data is kept in a MySQL database. It ended up being a rather larger application in the end, but really makes life quite a bit easier.