Glovebox: My Solution to Managing Servers
When I started in the “Technology” department at my current employer, I found myself apart of a team that was tasked with taking care of hundreds of IBM blade servers, and tens of other IBM system x servers. For the most part we could keep up with our servers by where they were in our monitoring software, but if we needed to know exactly where they were in either a blade center, by remote console name, or what Domain-0 they lived on for our Xen based virtual machines – we had to relate back to a usually out of date spreadsheet that showed where to go.
After a year of this mess I decided to go a different route – a dynamic web application that would update it self based on data pulled from the blade centers via SNMP. Glovebox was born. Well, actually it was called “the dashboard” in the beginning – unfortunately for me my co-workers have no issue pointing out when things aren’t named quite right, a sarcastic “well it ain’t a dashboard… how about Glovebox!” decided the name fate of it. The product took almost a year to get to its current self, but manages to keep track of all our blades, IBM RSA II consoles and our Xen virtual machines without the input from anyone on staff.
For the IBM blades and RSA II cards it uses a set of OIDs I gathered from sifting through their respective snmp output. In the case of the blades, those OIDs are:
| 188.8.131.52.184.108.40.206.220.127.116.11.1.1.3 | bladecenters | serial | | 18.104.22.168.22.214.171.124.126.96.36.199.1.1 | bladecenters | error | | 188.8.131.52.184.108.40.206.220.127.116.11.1.1.2 | bladecenters | family | | 18.104.22.168.22.214.171.124.126.96.36.199.1.1.1 | bladecenters | model | | 188.8.131.52.184.108.40.206.220.127.116.11.18.104.22.168 | switches | error | | 22.214.171.124.126.96.36.199.188.8.131.52.184.108.40.206 | switches | address | | 220.127.116.11.18.104.22.168.22.214.171.124.126.96.36.199 | servers | bios | | 188.8.131.52.184.108.40.206.220.127.116.11.18.104.22.168 | servers | serial | | 22.214.171.124.126.96.36.199.188.8.131.52.184.108.40.206 | servers | model | | 220.127.116.11.18.104.22.168.22.214.171.124.126.96.36.199 | servers | hostname | | 188.8.131.52.184.108.40.206.220.127.116.11.18.104.22.168 | servers | bsmp | | 22.214.171.124.126.96.36.199.188.8.131.52.184.108.40.206 | servers | error | | 220.127.116.11.18.104.22.168.22.214.171.124.126.96.36.199 | servers | power | | 188.8.131.52.184.108.40.206.220.127.116.11.18.104.22.168 | servers | storage | | 22.214.171.124.126.96.36.199.188.8.131.52.184.108.40.206 | servers | family | | 220.127.116.11.18.104.22.168.22.214.171.124.126.96.36.199 | switches | serial | | 188.8.131.52.184.108.40.206.220.127.116.11.2.1.2 | bladecenters | eventlog | | 18.104.22.168.22.214.171.124.126.96.36.199.1 | bladecenters | name | | 188.8.131.52.184.108.40.206.220.127.116.11.5.1.2 | bladecenters | watts | | 18.104.22.168.22.214.171.124.126.96.36.199.5.1.3 | bladecenters | btus | | 188.8.131.52.184.108.40.206.220.127.116.11.1.1.3 | bladecenters | hostname | | 18.104.22.168.22.214.171.124.126.96.36.199.1.2 | bladecenters | infoled | | 188.8.131.52.184.108.40.206.220.127.116.11.1.3 | bladecenters | templed | | 18.104.22.168.22.214.171.124.126.96.36.199.1.4 | bladecenters | identled | | 188.8.131.52.184.108.40.206.220.127.116.11.25 | bladecenters | installedblades | | 18.104.22.168.22.214.171.124.126.96.36.199.188.8.131.52 | mm | firmware | | 184.108.40.206.220.127.116.11.18.104.22.168.30 | bladecenters | installedmms | | 22.214.171.124.126.96.36.199.188.8.131.52.184.108.40.206 | mm | serial | | 220.127.116.11.18.104.22.168.22.214.171.124.126.96.36.199 | mm | firmware_date | | 188.8.131.52.184.108.40.206.220.127.116.11.18.104.22.168 | mm | name | | 22.214.171.124.126.96.36.199.188.8.131.52.1.1.3 | mm | address | | 184.108.40.206.220.127.116.11.18.104.22.168.1.1.5 | mm | error | | 22.214.171.124.126.96.36.199.188.8.131.52.1.1.4 | mm | primary |
From the list you can see I monitor not just the blades but also the management modules, the chassis, and the blade switches. The snmp output on the IBM Advanced Management Modules is quite complete and they provide good MIBs to be able to decipher the information.
I have a similar list for the RSA II cards which was added after a re-write of the initial version of Glovebox mid last year, the re-write was to make the software be able to use “modules” for each device, this was so adding future additions was easy. The Xen virtual machines are added via libvirt and its associated Perl module, I have setup keys between the server that this application runs on and each Domain-0, allowing it connection and figure out whats running on there. Since it based off modules you very well could add anything you wanted – it even has a shared development libary that allows for easy additions without re-inventing the wheel.
So whats it look like? Well it looks like any application written with ExtJS as the front end – quite good. I’m not a designer, I can’t do graphics, but with the help of ExtJS it came out quite nicely. How about a peak:
Sorry for blocking out some of the data, but some of the info needs to stay in house. The back end is entirely Perl and all data is kept in a MySQL database. It ended up being a rather larger application in the end, but really makes life quite a bit easier.