I specialize in one specific brand of MOM so I can't provide a complete answer. However, after 3 days there are no other answers so I though I'd at least take a stab at this.
What are some of the ones out there and which one[s] have the most unique features? [Multicast, p2p, alternative message structures, device detection, etc]
I'm not sure what you mean by "device detection" but WebSphere MQ has the rest of the things mentioned. There's also a REST API, file-to-message, message-to-file, rich user-defined message metadata capability, 1-phase commit, 2-phase commit, tunneling over HTTPS, encryption on the wire, encryption at rest, API's in a dozen languages, support for dozens of platforms, automatic failover and reconnect, broker-based message filtering, etc., etc.
And that's just the one product I'm familiar with. The messaging space is quite rich with competing products and some are purpose-built with narrow focus and others are general-purpose solutions. Some are lightweight in-memory affairs and some are quite robust. Sorry I can't provide a detailed comparative analysis, though. I was rather hoping someone would, or link to a good and recent overview article.
Also, are any of these systems scalable enough for heavy use on a grid or large amounts of data?
Oh yeah. The offerings range from ultra safe (messages hardened to disk & replicated while app blocks) to ultra fast (messages sprayed over multicast at a rate of thousands or tens of thousands per second). I have customers that move thousands of messages per second over a few nodes and others that in aggregate move hundreds of thousands of messages per second over a large network.