Use Hyperic HQ to monitor ServiceMix 4

Recently I've been helping out Tom Purcell (co-worker) with a presentation for our Emerging Technologies for the Enterprise conference being held in Philadelphia on March 26 - 27th, 2009. The presentation is called "The Bus in the Clouds" and it focuses on how organizations can use Amazon web services (i.e. cloud computing) to provide a secure and cost effective means of integration for trading partners.

The presentation is based on a typical use case that Tom has described in his latest blog post. The piece I've been looking at is monitoring and managing ServiceMix 4 (SMX4) using Hyperic HQ. Hyperic HQ is an open source monitoring software designed to manage web applications and infrastructure. What's interesting about Hyperic HQ is that FUSE HQ, MULE HQ and SpringSource Application Management Suite are all based on Hyperic. Now one question you might ask is "why use Hyperic HQ when you can monitor ServiceMix 4 with FUSE HQ?" The simple answer is that we are focused on a "subscription free" solution. Plus, we wanted to use the latest version of Hyperic HQ (v.4.0.3) and FUSE HQ is currently based on v.3.2.

One thing I quickly found out is that the Auto-discovery capability of Hyperic HQ did not find SMX4 "out of the box". Well after some research I found out that SMX4 is NOT one of the 75 common software products and technologies supported by Auto-discovery. Well, if you think about it some more you can probably understand why :) Anyway, after quite a bit of research and some help from Roberto Rojas (co-worker) I finally was able to hook Hyperic HQ into SMX4 to monitor Camel routes (see Roberto's Camel Component for Amazon SQS blog post). It was not that difficult however, not obvious either so I decided to blog about it.

Installation and Configuration

1. Download and install ServiceMix 4
2. Download and install Hyperic HQ. If you're installing Hyperic on one machine then just download the HQ Server Package for your OS. The installer will install both the server and agent.

NOTE: You need to install a plugin for SMX4 so it is best to install both the server and agent in the same directory (e.g. /opt/hyperic or c:\Program Files\Hyperic HQ).

Hyperic Linux Install on CentOS
The following is an example of installing Hyperic using the linux installer:

[chariot@dom:~/downloads/hyperic-hq-installer] ./setup.sh
Unpacking JRE to temporary directory /tmp/jre
Initializing Hyperic HQ 4.0.3 Installation...
Loading taskdefs...
Taskdefs loaded
Choose which software to install:
1: Hyperic HQ Server
2: Hyperic HQ Agent
You may enter multiple choices, separated by commas.
1,2
HQ server installation path [default '/home/hyperic']:
/opt/hyperic
HQ agent installation path [default '/opt/hyperic']:

Loading install configuration...
Install configuration loaded.
Preparing to install...
Validating agent install configuration...
Validating server install configuration...
Checking server webapp port...
Checking server secure webapp port...
Checking server JRMP port...
Checking server JNP port...
Verifying admin user properties
Validating server DB configuration...
Installing the agent...
Looking for previous installation
Unpacking agent to: /opt/hyperic/agent-4.0.3...
Installing the agent JRE ...
Unpacking JRE x86-linux-glibc2-jre-1_5.tar.gz to: /opt/hyperic/agent-4.0.3...
Setting permissions on agent binaries...
Fixing line endings on text files...
--------------------------------------------------------------------------------
Installation Complete:
Agent successfully installed to: /opt/hyperic/agent-4.0.3
--------------------------------------------------------------------------------

You can now start your HQ agent by running this command:

/opt/hyperic/agent-4.0.3/bin/hq-agent.sh start

Installing the server...
Unpacking server to: /opt/hyperic/server-4.0.3...
Creating server configuration files...
Copying binaries and libraries to server installation...
Copying server configuration file...
Copying server control file...
Copying server binaries...
Copying server libs...
Setting up server database...

Now login to another terminal as root and execute this script:

/home/chariot/downloads/hyperic-hq-installer/installer-4.0.3/data/hqdb/tune-os.sh

This script sets up the proper shared memory settings to run the
built-in database.

Press Enter after you run the script to continue this installation.

/etc/sysctl.conf defined an adequate kernel.shmmax, not changing anything

Tuning completed successfully!

Setting up JDBC driver...
Copying database files...
Configuring database...
Starting repopulation of configuration table...
Waiting for built-in database to start (on port 9432)...
Starting built-in database...
Preparing database...
Waiting for server to stop...
Stopping built-in database...
Built-in database stopped.
Installing the server JRE ...
Unpacking JRE x86-linux-glibc2-jre-1_5.tar.gz to: /opt/hyperic/server-4.0.3...
Setting permissions on server binaries...
Fixing line endings on text files...
--------------------------------------------------------------------------------
Installation Complete:
Server successfully installed to: /opt/hyperic/server-4.0.3
--------------------------------------------------------------------------------


You can now start your HQ server by running this command:

/opt/hyperic/server-4.0.3/bin/hq-server.sh start

Note that the first time the HQ server starts up it may take several minutes
to initialize. Subsequent startups will be much faster.

Once the HQ server reports that it has successfully started, you can log in
to your HQ server at:

http://localhost:7080/
username: hqadmin
password: hqadmin

To change your password, log in to the HQ server, click the "Administration"
link, choose "List Users", then click on the "hqadmin" user.


Setup completed.
A copy of the output shown above has been saved to:
/home/chariot/downloads/hyperic-hq-installer/installer-4.0.3/./hq-install.log

Deleting temporary JRE
[chariot@dom:~/downloads/hyperic-hq-installer]

3. Start the server first (e.g. /<hyperic directory>/<server directory>/bin/hq-server.sh start)
4. Wait for the server to start then start the agent (e.g. /<hyperic directory>/<agent directory>/bin/hq-agent.sh start)
5. Confirm that you can log into the console (http://localhost:7080/). You should see an Auto-discovery portal on the dashboard that has discovered some Hyperic servers/services.
6. Shutdown the server (e.g./<hyperic directory>/<server directory>/bin/hq-server.sh stop) and agent (e.g. /<hyperic directory>/<agent directory>/bin/hq-agent.sh stop)
7. Create the directory /<hyperic directory>/hq-plugins
8. Copy the servicemix-plugin.xml file to the /<hyperic directory>/hq-plugins directory.
9. Start both the server and agent
10. Log into the Hyperic console http://localhost:7080/
11. On the Dashboard click the Add to Inventory button in the Auto-Discovery portlet. This isn't mandatory but it will give you a chance to see how other services are monitored. For some unknown reason ServiceMix4 will NOT be listed in Auto-discovery. Therefore you have to add it manually.
12. Click on the Resources tab
13. Click on Platforms link then click on the server listed (should only be one)
14. Click on the Inventory tab
15. Scroll down to the Servers area and click on the New link
16. Enter in the following information:

Name: <some unique name> e.g. ServiceMix 4
Description: <a description>
Server Type: select 'ServiceMix 4.x' from the dropdown list
Install Path: <path to the root directory where SMX4 is installed>

17. Save the new server and you should see a ServiceMix server listed

Note: After manually adding the ServiceMix4 server you may be asked (presented a link) to configure the properties for the server. Click on the properties link and enter in the JMX password for servicemix (default password is 'smx') then save your changes.

The servicemix-plugin.xml file can be modified to monitor other areas of SMX4. Before going down that path I recommend that you read through the Hyperic documentation to get an understanding on how plugins work.

Hopefully I didn't miss anything...let me know if I did.