EAI/middleware: Huge messages inline or by reference

Question

Are there good practices for processing messages or requests/responses dealing with huge amounts of data (MBytes? GBytes?) via an middleware or an EAI system, assunming that the middleware transports but does not process the huge data blobs?

• Transport the data with the message in order to keep coherence but probably several times copy huge amounts of data between systems

• Keep the data in a store or data repository (file system, database,...) and transport a reference to the data, however loosing the close coupling between the message and the data

What is the common solution to this problem? How is this supported by EAI or middleware products?

Answer 1

If it is practical to make a direct connection between the source and destination system then sending only a reference through a message to start the transfer will work but, as you noted you lose the loose coupling.

According to the book Enterprise Integration Patterns you could implement the Message sequence pattern.

In my experience with EAI systems such as TIBCO. It pretty much up to you to figure our how to handle arbitrarily large amounts of data.