Monday, March 9, 2009

Use Hyperic HQ to monitor ServiceMix 4

Recently I've been helping out Tom Purcell (co-worker) with a presentation for our Emerging Technologies for the Enterprise conference being held in Philadelphia on March 26 - 27th, 2009. The presentation is called "The Bus in the Clouds" and it focuses on how organizations can use Amazon web services (i.e. cloud computing) to provide a secure and cost effective means of integration for trading partners.

The presentation is based on a typical use case that Tom has described in his latest blog post. The piece I've been looking at is monitoring and managing ServiceMix 4 (SMX4) using Hyperic HQ. Hyperic HQ is an open source monitoring software designed to manage web applications and infrastructure. What's interesting about Hyperic HQ is that FUSE HQ, MULE HQ and SpringSource Application Management Suite are all based on Hyperic. Now one question you might ask is "why use Hyperic HQ when you can monitor ServiceMix 4 with FUSE HQ?" The simple answer is that we are focused on a "subscription free" solution. Plus, we wanted to use the latest version of Hyperic HQ (v.4.0.3) and FUSE HQ is currently based on v.3.2.

One thing I quickly found out is that the Auto-discovery capability of Hyperic HQ did not find SMX4 "out of the box". Well after some research I found out that SMX4 is NOT one of the 75 common software products and technologies supported by Auto-discovery. Well, if you think about it some more you can probably understand why :) Anyway, after quite a bit of research and some help from Roberto Rojas (co-worker) I finally was able to hook Hyperic HQ into SMX4 to monitor Camel routes (see Roberto's Camel Component for Amazon SQS blog post). It was not that difficult however, not obvious either so I decided to blog about it.

Installation and Configuration

1. Download and install ServiceMix 4
2. Download and install Hyperic HQ. If you're installing Hyperic on one machine then just download the HQ Server Package for your OS. The installer will install both the server and agent.

NOTE: You need to install a plugin for SMX4 so it is best to install both the server and agent in the same directory (e.g. /opt/hyperic or c:\Program Files\Hyperic HQ).

Hyperic Linux Install on CentOS
The following is an example of installing Hyperic using the linux installer:

[chariot@dom:~/downloads/hyperic-hq-installer] ./setup.sh
Unpacking JRE to temporary directory /tmp/jre
Initializing Hyperic HQ 4.0.3 Installation...
Loading taskdefs...
Taskdefs loaded
Choose which software to install:
1: Hyperic HQ Server
2: Hyperic HQ Agent
You may enter multiple choices, separated by commas.
1,2
HQ server installation path [default '/home/hyperic']:
/opt/hyperic
HQ agent installation path [default '/opt/hyperic']:

Loading install configuration...
Install configuration loaded.
Preparing to install...
Validating agent install configuration...
Validating server install configuration...
Checking server webapp port...
Checking server secure webapp port...
Checking server JRMP port...
Checking server JNP port...
Verifying admin user properties
Validating server DB configuration...
Installing the agent...
Looking for previous installation
Unpacking agent to: /opt/hyperic/agent-4.0.3...
Installing the agent JRE ...
Unpacking JRE x86-linux-glibc2-jre-1_5.tar.gz to: /opt/hyperic/agent-4.0.3...
Setting permissions on agent binaries...
Fixing line endings on text files...
--------------------------------------------------------------------------------
Installation Complete:
Agent successfully installed to: /opt/hyperic/agent-4.0.3
--------------------------------------------------------------------------------

You can now start your HQ agent by running this command:

/opt/hyperic/agent-4.0.3/bin/hq-agent.sh start

Installing the server...
Unpacking server to: /opt/hyperic/server-4.0.3...
Creating server configuration files...
Copying binaries and libraries to server installation...
Copying server configuration file...
Copying server control file...
Copying server binaries...
Copying server libs...
Setting up server database...

Now login to another terminal as root and execute this script:

/home/chariot/downloads/hyperic-hq-installer/installer-4.0.3/data/hqdb/tune-os.sh

This script sets up the proper shared memory settings to run the
built-in database.

Press Enter after you run the script to continue this installation.

/etc/sysctl.conf defined an adequate kernel.shmmax, not changing anything

Tuning completed successfully!

Setting up JDBC driver...
Copying database files...
Configuring database...
Starting repopulation of configuration table...
Waiting for built-in database to start (on port 9432)...
Starting built-in database...
Preparing database...
Waiting for server to stop...
Stopping built-in database...
Built-in database stopped.
Installing the server JRE ...
Unpacking JRE x86-linux-glibc2-jre-1_5.tar.gz to: /opt/hyperic/server-4.0.3...
Setting permissions on server binaries...
Fixing line endings on text files...
--------------------------------------------------------------------------------
Installation Complete:
Server successfully installed to: /opt/hyperic/server-4.0.3
--------------------------------------------------------------------------------


You can now start your HQ server by running this command:

/opt/hyperic/server-4.0.3/bin/hq-server.sh start

Note that the first time the HQ server starts up it may take several minutes
to initialize. Subsequent startups will be much faster.

Once the HQ server reports that it has successfully started, you can log in
to your HQ server at:

http://localhost:7080/
username: hqadmin
password: hqadmin

To change your password, log in to the HQ server, click the "Administration"
link, choose "List Users", then click on the "hqadmin" user.


Setup completed.
A copy of the output shown above has been saved to:
/home/chariot/downloads/hyperic-hq-installer/installer-4.0.3/./hq-install.log

Deleting temporary JRE
[chariot@dom:~/downloads/hyperic-hq-installer]

3. Start the server first (e.g. /<hyperic directory>/<server directory>/bin/hq-server.sh start)
4. Wait for the server to start then start the agent (e.g.
/<hyperic directory>/<agent directory>/bin/hq-agent.sh start)
5. Confirm that you can log into the console (http://localhost:7080/). You should see an Auto-discovery portal on the dashboard that has discovered some Hyperic servers/services.
6. Shutdown the server (e.g.
/<hyperic directory>/<server directory>/bin/hq-server.sh stop) and agent (e.g. /<hyperic directory>/<agent directory>/bin/hq-agent.sh stop)
7. Create the directory
/<hyperic directory>/hq-plugins
8. Copy the servicemix-plugin.xml file to the
/<hyperic directory>/hq-plugins directory.
9. Start both the server and agent
10. Log into the Hyperic console http://localhost:7080/
11. On the Dashboard click the Add to Inventory button in the Auto-Discovery portlet. This isn't mandatory but it will give you a chance to see how other services are monitored. For some unknown reason ServiceMix4 will NOT be listed in Auto-discovery. Therefore you have to add it manually.
12. Click on the Resources tab
13. Click on Platforms link then click on the server listed (should only be one)
14. Click on the Inventory tab
15. Scroll down to the Servers area and click on the New link
16. Enter in the following information:

Name:
<some unique name> e.g. ServiceMix 4
Description:
<a description>
Server Type: select 'ServiceMix 4.x' from the dropdown list
Install Path:
<path to the root directory where SMX4 is installed>

17. Save the new server and you should see a ServiceMix server listed

Note: After manually adding the ServiceMix4 server you may be asked (presented a link) to configure the properties for the server. Click on the properties link and enter in the JMX password for servicemix (default password is 'smx') then save your changes.

The servicemix-plugin.xml file can be modified to monitor other areas of SMX4. Before going down that path I recommend that you read through the Hyperic documentation to get an understanding on how plugins work.

Hopefully I didn't miss anything...let me know if I did.

10 comments:

Anonymous said...

Rod,

As for the 'auto-discovery' I ran into this last year with Servicemix3.2 The servicemix.xml file in the plugins directory needs to be tweak. In our case, the plugin was for Servicemix 3.1. We just needed to update it to Service 3.2 and we were good to go. Keep in mind there are plugin .xml files in both the server folders as well as the agent folders.

Things can also get a little tricky if you are running multiple services that you want to auto-discover or monitor on the same machine. In our environment, we had 3 ActiveMQ servers and 3 SMX servers we wanted to monitor. You can achieve this by creating 'unique strings' in the start up commands of each services and the querying on that within Hyperic.

The Open version is good but for large enterprises the automation add-ins are well worth it. For example, server threshold notifications are set individually. For a few services this is OK. For a hundred services the time spent would easily justify the Enterprise version.

All in all, its a great product and I recommend it!

-john
john [at] tenui.com

Anonymous said...

Here is the line in the plugin.xml for servicemix we needed to fix:

server name="ServiceMix" version="3.x"

I haven't tried it yet in 4.x but we needed this to match our version in our servicemix.xml in the servicemix configuration files.

Again, this fixed needed to be put into both the agent plugin and the server plugin.

-john

john [at] tenui.com

Rod Biresch said...

John,
Thanks for sharing!

Yes, I did discover that there are plugin folders nested in both the server and agent directories. According to the Hyperic documentation the /<hyperic directory>/hq-plugins directory is a hot-deployment directory and the other plugin directories are now (as of version 4.0) considered alternate locations.

I'll take a closer look at matching the server version in the plugin xml to see if auto-discovery will work for SMX4.

It is a great product.

Thanks again!
Rod

Unknown said...

Hi ,
i installed hyperic hq open source edition Version 4.2.0.7 on BOSS(a Debian Derived Linux).

i ve succesfully installed both the server n agent on my local machine.

The problem i am not getting any graphs instead dat place is simply empty. I ve searched but unable to find solution.

Also , if i want to monitor other host with hyperic hq, what should be done??

the question may be silly, but I am unable to recover from this struck.

thanks a lot ....

Rod Biresch said...

Hello dvd, unfortunately I haven't kept up with monitoring ServiceMix with Hyperic. I spoke to a colleague at Chariot Solutions and he recommended looking at the Hyperic-based FUSE HQ for monitoring ServiceMix. FUSE HQ will likely have more ServiceMix specific features over OSS Hyperic. Good luck.

Unknown said...

Hi, I followed your instructions and I was able to install, however, after adding servicemix to the list of servers, it gave me this error:

The configuration has not been set for this resource due to : Invalid configuration: Error retrieving value: Failed to invoke getProcMem[State.Name.eq=java,Args.*.ct=servicemix.base]: Query did not match any processes: State.Name.eq=java,Args.*.ct=servicemix.base

What would cause this?

Rod Biresch said...

Hello Keizer! Unfortunately, as I mentioned previously, I haven't been current on the latest from ServiceMix and Hyperic. What versions are you using? If I get a chance I'll take a look to see if I get the same error.

Thanks,
Rod

Anonymous said...

Hi,
I am still not able to get Hyperic HQ to identify my services on servicemix 4. Has anybody being able to do this with SMX 4?

Anonymous said...

Hello,

I've just added this plugin using servicemix 4.4.0 and had to change both the jmx string to that set in the servicemix configuration and the

State.Name.eq=java,Args.*.ct=servicemix.base

to

State.Name.eq=java,Args.*.ct=karaf.base

HTH.

Unknown said...

I've been looking into several different application monitoring softwares and found this really interesting. Thanks!