Use Hyperic HQ to monitor ServiceMix 4

Recently I've been helping out Tom Purcell (co-worker) with a presentation for our Emerging Technologies for the Enterprise conference being held in Philadelphia on March 26 - 27th, 2009. The presentation is called "The Bus in the Clouds" and it focuses on how organizations can use Amazon web services (i.e. cloud computing) to provide a secure and cost effective means of integration for trading partners.

The presentation is based on a typical use case that Tom has described in his latest blog post. The piece I've been looking at is monitoring and managing ServiceMix 4 (SMX4) using Hyperic HQ. Hyperic HQ is an open source monitoring software designed to manage web applications and infrastructure. What's interesting about Hyperic HQ is that FUSE HQ, MULE HQ and SpringSource Application Management Suite are all based on Hyperic. Now one question you might ask is "why use Hyperic HQ when you can monitor ServiceMix 4 with FUSE HQ?" The simple answer is that we are focused on a "subscription free" solution. Plus, we wanted to use the latest version of Hyperic HQ (v.4.0.3) and FUSE HQ is currently based on v.3.2.

One thing I quickly found out is that the Auto-discovery capability of Hyperic HQ did not find SMX4 "out of the box". Well after some research I found out that SMX4 is NOT one of the 75 common software products and technologies supported by Auto-discovery. Well, if you think about it some more you can probably understand why :) Anyway, after quite a bit of research and some help from Roberto Rojas (co-worker) I finally was able to hook Hyperic HQ into SMX4 to monitor Camel routes (see Roberto's Camel Component for Amazon SQS blog post). It was not that difficult however, not obvious either so I decided to blog about it.

Installation and Configuration

1. Download and install ServiceMix 4
2. Download and install Hyperic HQ. If you're installing Hyperic on one machine then just download the HQ Server Package for your OS. The installer will install both the server and agent.

NOTE: You need to install a plugin for SMX4 so it is best to install both the server and agent in the same directory (e.g. /opt/hyperic or c:\Program Files\Hyperic HQ).

Hyperic Linux Install on CentOS
The following is an example of installing Hyperic using the linux installer:

[chariot@dom:~/downloads/hyperic-hq-installer] ./setup.sh
Unpacking JRE to temporary directory /tmp/jre
Initializing Hyperic HQ 4.0.3 Installation...
Loading taskdefs...
Taskdefs loaded
Choose which software to install:
1: Hyperic HQ Server
2: Hyperic HQ Agent
You may enter multiple choices, separated by commas.
1,2
HQ server installation path [default '/home/hyperic']:
/opt/hyperic
HQ agent installation path [default '/opt/hyperic']:

Loading install configuration...
Install configuration loaded.
Preparing to install...
Validating agent install configuration...
Validating server install configuration...
Checking server webapp port...
Checking server secure webapp port...
Checking server JRMP port...
Checking server JNP port...
Verifying admin user properties
Validating server DB configuration...
Installing the agent...
Looking for previous installation
Unpacking agent to: /opt/hyperic/agent-4.0.3...
Installing the agent JRE ...
Unpacking JRE x86-linux-glibc2-jre-1_5.tar.gz to: /opt/hyperic/agent-4.0.3...
Setting permissions on agent binaries...
Fixing line endings on text files...
--------------------------------------------------------------------------------
Installation Complete:
Agent successfully installed to: /opt/hyperic/agent-4.0.3
--------------------------------------------------------------------------------

You can now start your HQ agent by running this command:

/opt/hyperic/agent-4.0.3/bin/hq-agent.sh start

Installing the server...
Unpacking server to: /opt/hyperic/server-4.0.3...
Creating server configuration files...
Copying binaries and libraries to server installation...
Copying server configuration file...
Copying server control file...
Copying server binaries...
Copying server libs...
Setting up server database...

Now login to another terminal as root and execute this script:

/home/chariot/downloads/hyperic-hq-installer/installer-4.0.3/data/hqdb/tune-os.sh

This script sets up the proper shared memory settings to run the
built-in database.

Press Enter after you run the script to continue this installation.

/etc/sysctl.conf defined an adequate kernel.shmmax, not changing anything

Tuning completed successfully!

Setting up JDBC driver...
Copying database files...
Configuring database...
Starting repopulation of configuration table...
Waiting for built-in database to start (on port 9432)...
Starting built-in database...
Preparing database...
Waiting for server to stop...
Stopping built-in database...
Built-in database stopped.
Installing the server JRE ...
Unpacking JRE x86-linux-glibc2-jre-1_5.tar.gz to: /opt/hyperic/server-4.0.3...
Setting permissions on server binaries...
Fixing line endings on text files...
--------------------------------------------------------------------------------
Installation Complete:
Server successfully installed to: /opt/hyperic/server-4.0.3
--------------------------------------------------------------------------------


You can now start your HQ server by running this command:

/opt/hyperic/server-4.0.3/bin/hq-server.sh start

Note that the first time the HQ server starts up it may take several minutes
to initialize. Subsequent startups will be much faster.

Once the HQ server reports that it has successfully started, you can log in
to your HQ server at:

http://localhost:7080/
username: hqadmin
password: hqadmin

To change your password, log in to the HQ server, click the "Administration"
link, choose "List Users", then click on the "hqadmin" user.


Setup completed.
A copy of the output shown above has been saved to:
/home/chariot/downloads/hyperic-hq-installer/installer-4.0.3/./hq-install.log

Deleting temporary JRE
[chariot@dom:~/downloads/hyperic-hq-installer]

3. Start the server first (e.g. /<hyperic directory>/<server directory>/bin/hq-server.sh start)
4. Wait for the server to start then start the agent (e.g. /<hyperic directory>/<agent directory>/bin/hq-agent.sh start)
5. Confirm that you can log into the console (http://localhost:7080/). You should see an Auto-discovery portal on the dashboard that has discovered some Hyperic servers/services.
6. Shutdown the server (e.g./<hyperic directory>/<server directory>/bin/hq-server.sh stop) and agent (e.g. /<hyperic directory>/<agent directory>/bin/hq-agent.sh stop)
7. Create the directory /<hyperic directory>/hq-plugins
8. Copy the servicemix-plugin.xml file to the /<hyperic directory>/hq-plugins directory.
9. Start both the server and agent
10. Log into the Hyperic console http://localhost:7080/
11. On the Dashboard click the Add to Inventory button in the Auto-Discovery portlet. This isn't mandatory but it will give you a chance to see how other services are monitored. For some unknown reason ServiceMix4 will NOT be listed in Auto-discovery. Therefore you have to add it manually.
12. Click on the Resources tab
13. Click on Platforms link then click on the server listed (should only be one)
14. Click on the Inventory tab
15. Scroll down to the Servers area and click on the New link
16. Enter in the following information:

Name: <some unique name> e.g. ServiceMix 4
Description: <a description>
Server Type: select 'ServiceMix 4.x' from the dropdown list
Install Path: <path to the root directory where SMX4 is installed>

17. Save the new server and you should see a ServiceMix server listed

Note: After manually adding the ServiceMix4 server you may be asked (presented a link) to configure the properties for the server. Click on the properties link and enter in the JMX password for servicemix (default password is 'smx') then save your changes.

The servicemix-plugin.xml file can be modified to monitor other areas of SMX4. Before going down that path I recommend that you read through the Hyperic documentation to get an understanding on how plugins work.

Hopefully I didn't miss anything...let me know if I did.

SOA Governance Refcard

I'm a pragmatist and this blog focuses on integration/SOA technology so one might wonder why am I blogging about SOA governance? Well let me say this first, governance is a real challenge for companies and I've seen first hand how large corporations struggle with adoption and change that comes with SOA. When I'm consulting on SOA engagements, I find myself getting pulled into meetings and conversations about SOA governance. Early adopters struggle with not only what they need to do with governance but more importantly how to effectively introduce change in their organization. I came across a very good refcard by Todd Biske that clearly and concisely addresses the "what" in SOA Governance...I highly recommend it. "How" to implement governance in an organization is somewhat unique and doesn't lend itself to a refcard or template.

So, to answer my earlier question about why this post on this blog...SOA is an enterprise initiative and if you're working with SOA technology you will eventually be faced with governance. So, to have something in your back pocket to help others or simply refer to is a great asset.

Cloud Computing Event

I thought I'd pass this along...my company Chariot Solutions is hosting Fall Forecast 2008 - Computing Among The Clouds this October 17th. The 2008 Fall Forecast conference highlights the emerging areas of cloud computing, software as a service and multi-core technologies. What's most appealing to me about cloud computing is the ability to quickly stand up a computing platform with very little investment (both cost and time). This great solution for working with SOA/Integration products because they can demand significant computing resources.

RESTful Web Services using IONA FUSE ESB 4.0 Preview and Spring Dynamic Modules

Roberto Rojas has a very nice blog post on RESTful Web Services using IONA FUSE ESB 4.0 Preview and Spring Dynamic Modules, which I highly recommend reading!

What's new in ServiceMix 4.x?

Recently, I was talking with a client about a new project of theirs based on the latest release FUSE ESB 3.x. That got me thinking, I knew that a FUSE ESB 4.0 Preview / ServiceMix 4.x (SMX4) was near release so in my efforts to keep current (as well as inform our client) I decided to take a closer look. I was able to find pretty much information on SMX4 however, I wasn't able to find a concise write-up that touched on key changes between the versions. So after some research I decided to blog about what I found. In this post I'm only focusing on the major changes with this new version without getting too far into the “weeds”.

So, what's new in SMX4? This is the first question I asked when I heard that a new major release of ServiceMix is on the horizon. Like most people, I pulled down the latest binaries of SMX4 and started going through the documentation. However, first before jumping into SMX4 lets get a quick picture of what the ServiceMix 3.x (SMX3) technology stack looks like so we can get a point of reference.

In SMX3 the JBI 1.0 container is a runtime environment for a Normalized Message Router (NMR) and JBI components (e.g. Service Engines and Binding Components) leveraging Apache ActiveMQ as the Message Broker. Typically an ESB offers at least message routing (e.g. ServiceMix EIP & Apache Camel), transformation (e.g. XSLT) and connectivity (e.g. File, JMS, HTTP, etc.) as does SMX3. Apache CXF is used to provide SOAP support and xml marshaling. From an operations perspective, SMX3 provides JMX-based management for running components and other internals as well as run standalone or embedded in an application server or web container (e.g. JBoss, Tomcat, etc.). There is a lot more that SMX3 has to offer however for this post that's a good picture. One last note on SMX3, version 3.2 and higher requires Java 5.

In SMX4 the technology stack really hasn't changed much. The key technologies are still there like Apache ActiveMQ, Apache CXF, JMX, etc. Apache Camel is the preferred technology for routing however, ServiceMix EIP appears to still be supported. Ok, so what has changed?

Looking at the architecture, the most noticeable change is the separation of the JBI container into a ServiceMix Kernel and a ServiceMix NMR. The ServiceMix Kernel is a lightweight OSGi-based environment deployed on Apache Felix (an OSGi R4 Services Platform implementation). The kernel and NMR are their own projects in Apache and can be downloaded and built separately. However, not to worry, if you download FUSE 4 preview (certified release of SMX4) from IONA you get the kernel and NMR bundled along with other stuff to run the ESB.

Why the change to OSGi?

James Strachan (ServiceMix co-founder and committer) pointed out to me that one of JBI's weakest areas is the cumbersome classloader model and that needed to be addressed. OSGi solves this problem with advanced OSGi bundling and classloading mechanisms. Ok, that sounds good but what is OSGi? Well in a nutshell, it is an environment where bundles (modules) communicate through well-defined services while hiding their internals from other bundles. OSGi also specifies how components are installed and managed. Components can be started, stopped, updated and uninstalled dynamically without bringing down the entire server. This is very beneficial for high availability systems (like an ESB). OSGi bundles are versioned and only bundles that can collaborate are wired together in the same class space. This solves the problem with library dependencies or JAR hell :) There are many more benefits to OSGi and the OSGi Alliance is a great place to start learning. With this change, ServiceMix now becomes an OSGi and JBI-based ESB.

What changed in the NMR?

The obvious change is that the NMR is now it's own Apache project, ServiceMix NMR. So, theoretically you could run different versions of the SeviceMix Kernel and NMR. Unfortunately I don't have a good use case for that although, this separation does provide flexibility. The NMR is now a set of OSGi bundles that can be deployed on any OSGi runtime but mainly built on top of the ServiceMix Kernel. The NMR project is also the JBI 1.0 implementation with plans to support JBI 2.0 in the future. There are a few things missing with ServiceMix NMR 1.0-m1 release (from the release notes):

• no support for JMX deployment and ant tasks
• no support for Service Assemblies Connections
• no support for transactions (a transaction manager and a naming context can be injected into components if they are available as OSGi services, but not transaction processing - suspend / resume - will be performed, as it would be requested for real support)...more on this a little later.

So depending on what you're doing, these missing capabilities may or may not hamper early adoption.

A few other interesting changes to mention. First, SMX4 is now a standalone only runtime. Previous versions supported embedding in application servers, web containers as well as applications. Flows are no longer used for message exchanges. This confused me a little however, thanks to James he clarified things. Flows are NMR flow types that are the mechanism by which the NMR sends messages from one Binding Component or Service Engine to another. SMX3 has 4 flow types that ended up causing confusion and added complexity for developers. This was simplified in SMX4 with a tendency to use explicit routes by exploiting Camel's routing capabilities.

Last change to note, SMX4 no longer performs transaction specific processing. However, it will pass transaction information along as it would any other message. I was fortunate enough to receive some insight from Guillaume Nodet (Apache Project Management Committee Chair and active committer) regarding the changes in transaction support and it turns out that the release notes are a little misleading. Guillaume explained that previously the NMR was considered as a transactional resource, which means the sending of a JBI exchange through the NMR could be part of the transaction (depending if it was sent synchronously or not). This is no longer the case with SMX4. Transactions are now conveyed along with the exchanges whether they are sent synchronously or asynchronously. As a result, the NMR does not need to be aware of transactions because they are now controlled by the different components instead. The benefits from this change include:

• Improved scalability of ServiceMix when using transactions because all JBI exchanges are now sent asynchronously.
• Improved interoperability as the other JBI containers tend to follow the same pattern when/if they support transactions.

Hopefully that gives you a pretty good sense of what is new in SMX4.

Building a “SOApp” with Mule (Open Source ESB)

Ok, I know that we're all up to our ears with technical acronyms, terms, buzzwords, etc. that all appear to have some marketing spin to them in hope that they will eventually work their way into industry acceptance. So, why should this be any different...just kidding, kind of :) Anyway, bear with me through this first paragraph and you'll understand how this “SOApp” thing came into existence in my little world. In 2007, I was fortunate enough to be part of a development team at Chariot Solutions responsible for designing and developing a product for the Material Handling and Distribution Industry. After a few requirements gathering sessions with the client, we discovered that the project had a number of integration points with new and existing systems...the age old integration challenge. The existing systems ranged from a COTS warehouse management system to a C-based controller that interfaced with physical hardware devices (e.g. lights, switches, buttons, etc.). Steve Smith (Chariot Architect) was involved early on in the initial discovery sessions and was quick to point out that all these integration points (files, database tables, TCP messages, etc.) we were faced with are very similar to challenges faced while implementing an SOA but on a much smaller scale. So, why not take advantage of the benefits that an ESB provides (i.e. protocol translation, message transformation and routing) to speed development and deliver a maintainable solution? Hence the “SOApp” concept was born. Well, the term “SOApp” (Service Oriented Application) was a play on the SOA acronym about 3 months ago when I was considering writing up a post (this post) to discuss a unique way of using ESB technology. A close second was “Save Our A...”, well you get the picture, anyway, back to the interesting technical stuff.

Ok, so we had a conceptual architecture centered around an ESB. The team was all in agreement that we would benefit from NOT having to write custom integration code and simply focus on developing business logic in POJOs. The decision to use Mule (v. 1.4.1) for the ESB was an easy choice for us because we had plenty of experience using it in our SOA Lab. There were many interesting aspects of the project however, this post focuses on how we dealt with the various integration challenges. Let's take a closer look. The diagram below illustrates what the architecture looks like with a focus on various system interfaces.



Warehouse Management System (WMS) – File-based Interface
The WMS generated flat files in a proprietary format on some random schedule. This data is essential to the overall operation of the entire system. These flat files needed to be processed as soon as the would appear in a directory. A Mule file connector was configured to poll a directory, pickup a file, pass the data onto a POJO for processing and eventually update the PostgreSQL database via Hibernate. Nothing fancy here just basic file processing.

PC Controller – TCP Sockets Interface
This controller serves as a central integration point for various switches, sensors, etc. A basic use case was when a sensor was triggered, the controller would send a proprietary delimited length text message was sent to a particular POJO handling that transaction in Mule. The POJO would typically query the PostgreSQL database and return the resulting data back to the controller. There was a non-functional requirement for the entire transaction to take less 200 ms. At first, we used a Mule TCP connector to communicate (both inbound and outbound) with the controller however, due to some limitations on the controller side (and time constraints) we had to switch outbound communications to custom socket code. Here was a case where we only benefited from using Mule's connectors for one communications direction however, since the socket code was easy enough to write it didn't take long to develop a workaround. Oh, the transaction performance ended up to be less than 100ms on average.

Pick-To-Light Controller – JDBC Connector Interface
This controller was a java-based controller that interfaced physical devices via the PostgreSQL database. Though this is not the “ideal” method for interfacing it was a constraint that we had to deal with. What was interesting about this interface is that we configured a Mule JDBC connector to process incoming requests when information appeared in the database tables. The POJOs that processed the data then used our Hibernate persistence layer to respond back to the controller via database tables.

Java Swing Clients – MuleClient / TCP Interface
There were two Java Swing clients that we developed for administrators and end users to interface with the system. These swing clients used the MuleClient to simplify the TCP communications with Mule. Each POJO was considered a “service” with a unique endpoint deployed in Mule. What is interesting about this is that the swing client only had to specify a single uri (e.g. tcp://localhost:) along with a payload and let the MuleClient handle the rest. There was no need to identify a specific endpoint in the URI, which made this design very flexible and maintainable. Here we leveraged the content-based routing capabilities of Mule by configuring various FilteringOutboundRouter's to call internal POJOs for processing. The outbound routers inspect the payload type and routed accordingly based on configuration.

CLI (Command Line Interface) Java Application – Telnet & MuleClient / TCP Interface
Well I saved the most interesting interface for last, IMHO :) One of the requirements was to have a handheld Windows CE device, capable of scanning barcodes, perform CRUD operations on the PostgreSQL database over a wireless 802.11b network. We had part of this problem solved by using the MuleClient to communicate with our POJOs deployed in Mule. Ok, so, how do we implement the MuleClient on a Windows CE device? Well, John Shepard (Chariot Architect) came up with a clever design to use Telnet and a basic Java application to solve the problem. This was a great example of leveraging proven technology along with a creative way of configuring Unix accounts. The end user accounts were configured to start the CLI java application after a successful login to the Unix machine instead of a bash shell. This work very well and performed beautifully. There was no need to install and manage client applications or worry about performance of the handheld device. The built-in Telnet capability of the Windows CE device was all that was needed on the client. This was very cool to see in action especially knowing that only a basic java application was behind it.

The purpose of this post was to give you an overview of how to use an ESB in new and creative ways aside from their original intent. In the future, I may expand more on these interfaces to provide more detail. There were a number of other interesting aspects to the project like using annotations and XDoclet to automatically generate the Mule configuration, etc. that is good material for future posts. Wrapping this up, we did take on some risk of using Mule in this capacity, however, luckily there were no major problems we ran into. Besides we always had Java to fall back on if Mule came up short anywhere. Could this be the beginning of a new movement in application development...the “SOApp”? Maybe, however I can already hear the snickering of my colleges as they read the end of this post :)

Use JConsole with ActiveMQ for a quick JMS test client

As I was working with OpenESB to try an integrate it with ActiveMQ I stumbled across a quick and easy way work with ActiveMQ. My needs were basic and I just wanted to be able to browse, add and remove messages from a set of queues. I knew that ActiveMQ has JMX support so I wondered if my basic needs could be met using JConsole. Sure enough they were. Here's what I did:

First, I configured ActiveMQ with the queues I needed and started the server (console output below).


The JMX service connection uri is part of the ActiveMQ startup output (see the last line beginning with INFO ManagementContext). After the server is running you then can start up JConsole by typing jconsole at a terminal/command prompt. The ActiveMQ JMX agent should be displayed in the local tab of the JConsole: Connect to Agent dialog box. If not, you can enter in the management context uri (e.g. service:jmx:rmi:///jndi/rmi://localhost:1099/jmxrmi)


After connecting you will be presented with a Summary screen showing a snapshot of ActiveMQ. From here simply click on the MBeans tab and expand the org.apache.activemq node to see the queues and topics configured in ActiveMQ.



Ok, now we're able perform as many MBean operations that are available. When you're unit testing a JMS service (or application) the first thing you typically do is place a message on a queue. You're probably saying "well duh?" and that's what I'd say. The funny thing is that I find myself spending more time googling for a tool or writing/modifying sample test code to do this simple task. You'll be amazed at how easy it is through jconsole to place a text message on a queue. Again, select the queue you want to put a message on and then on the Operations tab in jconsole and scroll down to the operation sendTextMessage. Fill in the parameter with some text and click the sendTextMessage button.



You should see a dialog box pop up with the operation return and a message ID. Congratulations you just put a text message on a queue. You can now experiment with some of the operations like browse, purge, etc. There you have it...a simply easy to use JMS test client.

It's very easy to find the queues and topics in the MBeans tree with a stand-alone instance of ActiveMQ. However, it gets very difficult to even find the ActiveMQ node when you attach to an application server (e.g. Geronimo) that comes bundled with ActiveMQ. I don't have a good answer for that yet.

Happy messaging :)