• POC for other messaging systems

Assignee: MSS

Estimated time for POC: ActiveMQ 5 MD (Usikkerhed A), Webservice 7 MD (usikkerhed A). Bemærk at dette er estimaterne for Proof-Of-Concept, og disse er temmelig sikker, da scope er tænkt timebox'et.

Background

Alternatives to the current SunMQ JMS based communication framework could be:

• Maintain a message based communication structure, but switch to another broker.
• Switch to another communication paradigm altogether, this properly means some kind of webservice communication like REST.

Java Management Extensions (JMX) has also been mentioned at a possibility, but as the name implies the purpose of JMX is to provide management and monitoring capapilities, and the usage of JMX for remote procedure calls would be unnatural.

Problems with the Current Setup

SunMQ

We are in the current setup bound to Sun MQ implementation, which is not fully open source. The concern over this issue has further increased after Oracles accusation of Sun, with the following focus shift from open systems to proprietary products. As NAS is a open source project, we would very must like to avoid any dependency on proprietary software.

Asynchonous communication

We have in several instances been struggling with the asynchronous communication used in NetarchiveSuite. The problems can be separated into two categories:

• Need for synchronize communication: In case of the dispatching of harvest job, we would like to avoid buffering job in a message queue, and instead configure and send jobs to the harvester when they are ready (FR 1774 Stop using the JMS queues for queuing snapshot harvests).
• Avoid congestions in the system: Because the message buffering mechanism allows messaging producers to generate large amounts request to the message consumers, we can encounter it is difficult to control situations with high load, because no feedback mechanism exists that can stop the production of further requests. If synchronous means of communication existed, these could be used to block producers from sending more requests.

Advantages of a Shift to ActiveMQ

• We would switch to an open source messaging system

• ActiveMQ is the current message bus chosen for the Bitmagasin projects, which NetarchiveSuite at some point is going to use for bit storage. This means that we sooner or later are going to have to use the ActiveMQ message bus anyway.

Risks Associated with a Shift to ActiveMQ

• We would switch from a proven (in NAs) messaging infrastructure to a new message bus, this is bound to introduce some unforeseen problems.
• The current JMS code is SunMQ specific, a switch would mean spending time reimplementing some of the JMS functionality and properly introduce bugs. Som of these issues might be alleviated by reusing some of the message bus components, which is going to be part of the Bitmagasin system.
• The current messaging infrastructure is based on dynamically created queues and topics. This was the reason for not using ActiveMQ earlier in the project, as AqtiveMQ didn't support this at the time. This can be solved in two ways:
  • ActiveMQ now has support for dynamically created queues
  • Move to a model with statically created queues.

Advantages of a shift to webservices (REST)

• We would gain much better control of when requests where processed by the 'message consumers' or service providers as they would be called in a WS setup.
• We are not forced to change all communication to be WS based. We could chose to use WS where relevant and stick to MQ where a change would be unnecessary.
• We have examples of quite complex code attempting to handle

Risks associated with a shift to Webservices

• We would properly have to implement some buffering mechanisms in the 'message producers' or clients as they would be called in a WS setup.
• We would create a tight coupling between the clients and service providers. These are currently decouple via. the Message broker. We do already have a centralized configuration and startup of the system (so the decoupling isn't more strict than we do have global bindings), so a specific client-server binding could be defined here. We might also use the current JMX binding to connect the ws participants.
• Handling of client and service provider restarts: The buffering of requests on the message bus in the current system, does make the system robust against application restarts. A WS based communication solution would require this to be handled by hand. In both cases we have to handle losing requests though (eg. the message consumer is restarted while is is processing a request, in both cases this means the request needs to be resent), so the different isn't so big.

Conclusions

Switching to ActiveMQ as message infrastructure seems like a small step. The switch could be even easier, when Bitmagasin ActiveMQ specific libraries are available.

Adapting a WS based approach to request-response operations for some of the application communications might be a good idea, and we should consider doing a proof-of-concept investigating of this (perhaps using the dispatch harvest job operation, which is difficult to implement as specified inFR 1774 based on the current asynchronous mq infrastructure).