IBM MQ for file transfer

Question

We are facing a choice to use IBM MQ over SFTP for file transfer. I've heard advantages of such approach, but I've never see anyone actually using it for a large files.

So main question: how well IMB MQ can handle transfer of large files (up to 100 MB)? Is it stable? It's from mainframe to UNIX server, if it does matter.

Thanks a lot.

Answer 1

I've used MQ with files up to 8GB in size, without incident. You have to allocate enough space for MQ to manage them, but it works.

Answer 2

Vendors have been doing File transfer over a messaging system such as WebSphere MQ for over 12 years. I have not used the IBM product. Yet, I know until recently IBM resold Metastorm’s product (under the brand PM4Data) and only recently entered the market with a first generation product.

I’ve seen Metastorm Integration Manager handle files of terabyte sizes all of the time; breaking up the messages across WebSphere MQ and performing buffers acknowledgements so not to overflow message queues or having to do a lot of MQ Administration.

Other worthwhile considerations of using messaging systems for file transfer vs. SFTP include:

1. Pre-and Post-Data Movement Handling - As a network transport, SFTP moves data (as can file transfer products over WebSphere MQ). To do pre- or post-movement processing, you’d have to write your own monitoring, triggering, error handling, etc. Mature vendors in the file transfer over messaging space provide infrastructure out-of-the-box for you. Depending on your application, they can also help
2. Audit Trail - SFTP provides often provide system-level logging. Often time, vendors providing file integration over MQ have greater audit capabilities, including the Pre- and Post-Data Movement Handling. Some vendors also offer secure web-based systems to expose certain integrations to authorized users.
3. Exception Handling – SFTP will give synchronous error messages from which you then code your error logic. Vendors providing file integration over MQ often provide exception capabilities ranging from "here’s a message on queue signifying a problem happened... good luck" to built-in integration to Web-based consoles or even BPM system.
4. Treating File Transfer as a Service – Often times, File Transfer is part of a larger business process. Some vendors leverage this to chain file transfers and even have them collaborate with human-centric processes to provide end-to-end visibility around the context of a point-to-point file transfer. Additionally, using messaging systems for file transfer helps move an organization to a more real-time, message-driven approach.

I hope the additional details give some details in your assessment. Good luck!

Answer 3

MQ itself offers message-based communication between programs. You can't use it directly to transfer files. People talking about "sending files via MQ" often mean that they will write the programs themselves to read from a file, send over MQ, receive from MQ and write to another file. This sounds temptingly trivial at first but quickly becomes very complex if you want it to be robust. Not recommended.

What you should be doing if you want to involve MQ in your file transfers is use one of the file-transfer products that sit upon MQ. There are a number of mature offerings, about which I know next to nothing, together with a new product from IBM itself ("Websphere MQ File Transfer Edition") which is being actively developed and recently announced an updated version (7.0.1).

To answer your specific question, I've personally used WMQFTE to transfer a file of 1000GB from an AIX machine to a Windows one, so I don't think your "large" 100MB files will be a problem :-). I don't get involved with the mainframe as much but it is very much a core platform for FTE.

Answer 4

In one of the previous answers, there is the followng statement:

People talking about "sending files via MQ" often mean that they will write the programs themselves to read from a file, send over MQ, receive from MQ and write to another file. This sounds temptingly trivial at first but quickly becomes very complex if you want it to be robust. Not recommended.

In theory it would be nice to "sending files via MQ" because you would receive a Confirmation on Delivery (CoD) message when the recipient got (read) the file from the queue. Another benefit would be that the file would be automatically removed from the queue when the recipient got (read) it from the queue.

Do you perhaps know whether the above (or a similar) is offered by MQFTE?

Answer 5

Capitalwares MQ File Mover is an option here.

Also WebSphere Message Broker for Remote Adapter Deployment and WebSphere MQ File Transfer Edition are options to consider.

Answer 6

T.Rob,

Thanks for the answer.