Since the dawn of time in EAI, designers of messaging systems have encouraged an arrangement of queues or topics that is primarily content-oriented. Meanwhile, most real EAI contexts are susceptible to problems arising from different but related messages being processed in the wrong order. A content-oriented topology ignores this and even exacerbates it. Is a practice established over decades actually more of an anti-pattern?
Keeping orders in order
My EAI example has a pretend order management system (P-OMS) with some message queues connecting it to the equally imaginary e-commerce portal (EI-ECP):
I have fictitiously interacted with my system as follows:
- Placed an order (on the Orders queue)
Realised that I won’t be at home when the order is due to be delivered
- Cancelled the order
- Changed the delivery address to that of the office (on the Customers queue)
- Placed the order again
And it’s worked! Great – that was lucky. Lucky? Well, lucky because there’s absolutely nothing in my system that guarantees the order will be delivered to the right place.
Equally legitimately, the P-OMS could have processed the messages in the sequence
- New order
- Address change
and so dispatched my order to the original address.
The state of the art
So this is a pretty basic problem. We’re talking about the requirement to preserve the order of events so that each participating application sees things occurring in the same sequence. As such, and given that messaging has been around for decades (excuse me a moment while I reminisce about the marvellous Stratus VOS messaging of the mid-1980s) best practice will have been established and products tweaked to support this, right?
Well, let’s have a look:
“the messaging system has different Message Channels for different types of information the applications want to communicate” (Enterprise Integration Patterns, p61)
“published messages are characterized into classes” “subscribers express interest in one or more classes” (Wikipedia, Publish-subscribe pattern)
“JMS clients publish messages to, and subscribe to messages from, a well-known node in a content-based hierarchy. JMS calls these nodes topics.” (JMS 2.0 specification, section 4.2.2)
Despite the slightly differing terminology, it seems that these designers all support – promote, even – a model where queues or topics are based primarily on content type. And, by a large majority, organizations that I’ve seen have embraced this advice, often going so far as to have separate queues for the equivalent of orders vs. cancellations, new customers vs. customer updates etc. based on the message structures associated with each event.
It doesn’t seem an exaggeration to suggest that this is inviting trouble – trouble of that awkward kind that doesn’t always surface in testing. Some organizations will notice a potential problem and implement a “resequencer” that is able to wait for related messages and sort them before processing. Although awkward, this will work where related messages are expected. In scenarios like the one above, however, nothing can anticipate the lurking customer update.
One could speculate on how this situation has arisen, specifically, why it is that we can’t define channels and topics separately and then relate them. I’m pretty sure it’s down to the major products originating in contexts far removed from the “corporate transactional data” world in which they’re used today. However, it’s perhaps more productive to look at a solution, and one that’s particularly easy to implement with our favourite non-corporate messaging service, RabbitMQ.
Rabbit to the rescue
To introduce publish-subscribe to the pretend OMS scenario I’ll add a fanciful CRM system that’s also interested in customer updates:
In RabbitMQ, we normally publish to an exchange rather than directly to a queue. Exchanges don’t store messages themselves, they simply route (and copy, if required) messages to destination queues as determined by the type of exchange and the routing logic. Many valid solutions to the order-preserving requirement are possible using combinations of (possibly multiple) exchanges, queue bindings and message properties. The one I’ll describe below combines a source-oriented exchange with message-type headers.
Rather than a “content type”-oriented topic or exchange name, we choose one related to the source system as it is the source system’s potential to generate a related sequence of messages that we want to capture. The EI-ECP system might be a specific product called SupaShop or something but this isn’t a great name for an exchange as it is obviously specific to the implementation. Instead we choose something that represents its role in the abstract, here just “the ECP”.
The queues are then named after the source and qualified by their destination – OMS or CRM – again using abstract app names.
Given that the ECP sends both customer and order messages to the same exchange which must then route them, the exchange needs the message type to be visible. Although we could perfectly well use a Topic Exchange and pass the type as a routing key, it is slightly more flexible to use a Headers Exchange and carry the type name as a simple header (a header without a value).
For those following at home
To walk through this on the management console:
1) Add the ECP Headers Exchange
2) Add the queue for CRM
3) Add a binding for the queue that matches Customer-related messages. The headers exchange recognises the x-match = any arg as telling it to match any of the headers listed. (The similar t- prefix for the types is arbitrary).
4) Add the queue for OMS
5) Add a binding to select both the Customer and Order types
6) Exercise the configuration by sending a CreateOrder message – this should only go to the OMS
7) A Customer update should go to both OMS and CRM
What’s the equivalent of this RabbitMQ arrangement in other messaging systems such as JMS? For those few JMS publish-subscribe systems supporting truly durable topic subscribers (subscriptions that can be created administratively) the JMS Message Selector can be defined to filter the message types of interest. This filtering will then be done in the provider, before the message is sent to the subscriber.
For other JMS publish-subscribe systems, a durable subscription might be created by connecting a dedicated queue as a subscriber to a topic. Hitching a queue to a topic isn’t a standard JMS feature but can be done via a Service Bus or similar routing agent (Mule, Spring Integration etc). The routing agent logic performs the same function as the bindings created above for RabbitMQ queues, either using JMS Message Selectors again or otherwise detecting the headers corresponding to the message types wanted for each queue.