Best Publish/Subscribe "Middleware"

Question

I'm in the market for a good open source network based Pub/Sub (observer pattern) library. I haven't found any I like:

• JMS - tied to Java, treats message contents as dumb binary blobs

• NDDS - $$, use of IDL

• CORBA/ICE - Pub/Sub is built on-top of RPC, CORBA API is non-intuitive

• JBOSS/ESB - not too familiar with

It would be nice if such a package could to the following:

• Network based

• Aware of payload data, users should not have to worry about endian/serialization issues

• Multiple language support (C++, ruby, Java, python would be nice)

• No auto-generated code (no IDLs!)

• Intuitive subscription/topic management

For fun, I've created my own. Thoughts?

Answer 1

You might want to look into RabbitMQ.

Answer 2

We use the RTI DDS implementation. It costs $$, but it supports many quality of service parameters.

There is a free DDS implementation called OpenDDS, but I've not used it.

I don't see how you can get around the need to predefine your data types if the target language is statically typed.

Answer 3

Look a bit deeper into the various JMS implementations.

Most of them are not Java only, they provide client libraries for other languages too.

Suns OpenMQ have at least a C++ interface, Apache ActiveMQ provides client side libraries for many common languages.

When it comes to message formats, they're usually decoupled from the message middleware itself. You could define your own message format. You could define your own XML schema and send XML messages. You could send BER encoded ASN.1 using some 3. party library if you want. Or format and parse the data with a JSON library.

Answer 4

You might take a look at PubSubHubBub. It's a extension to Atom/RSS to alow pubsub through webhooks. The interface is HTTP and XML, so it's language-agnostic. It's gaining increasing adoption now that Google Reader, FriendFeed and FeedBurner are using it. The main use case is blogs and stuff, but of course you can have any sort of payload.

The only open source implementation I know of so far is this one for the Google AppEngine. They say support for self-hosting is coming.

Answer 5

You might be interested in the MUSCLE library (disclaimer: I wrote it, so I may be biased). I think it meets all of the criteria you specified.

https://public.msli.com/lcs/muscle/

Answer 6

There is also OpenSplice DDS. This one is similar to RTI's DDS, except that it's LGPL!

Check it out:

Answer 7

As pointed-out by an earlier post in this thread, one of your options is OpenSplice DDS which is an Open Source implementation of the OMG DDS Standard (the same standard implemented by NDDS).

The main advantages of OpenSplice DDS over the other middleware you are considering can be summarized as:

• Performance
• Rich support for QoS (Persistence, Fault-Tolerance, Timeliness, etc.)
• Data Centricity (e.g. possibility of querying and filtering data streams)

Something that I'd like to understand is what are your issues with IDL. DDS uses IDL as language-independent way of specifying user data types. However DDS is not limited to IDL, you could be using XML, if you prefer. The advantage of specifying your data types, and decoupling their representation from a specific programming language, is that the middleware can:

(1) take away from you the burden of serializing data,

(2) generate very time/space efficient serialization,

(3) ensure end-to-end type safety,

(4) allow content filtering on the whole data type (not just the header like in JMS), and

(5) enable on-the wire interoperability across programming languages (e.g. Java, C/C++, C#, etc.)

Depending on the system or application you are designing, some of the properties above might not be useful/relevant. In that case, you can simply generate one, a few, "DDS Type" which is the holder of you serialized data.

If you think about JMS, it provides you with 5 different topic types you can use to send your data. With DDS you can do the same, but you have the flexibility to define exactly the topic types.

Finally, you might want to check out this blog entry on Scala and DDS for a longer discussion on why types and static-typing are good especially in distributed systems.

-AC

Answer 8

Three I've used:

• IBM MQ Series - Too Expensive, hard to work with.

• Tico Rendezvous - (renamed now to EMS?) Was very fast, used UDP, could also be used with no central server. My favorite but expensive and requires a maint fee.

• ActiveMQ - I'm using this currently but finding it crashes frequently. Also it's requires some projects ported from Java like spring.net. It works but I can't recommend it due to stability issues.

Also used MSMQ in an attempt to build my own Pub/Sub, but since it doesn't handle it out of the box your stuck writing a considerable amount of code.