Large amount of data - what is the best way to send them?

Question

Hi, we have this scenario:

A server which contains needed data and client component which these data wants.

On the server are stored 2 types of data: - some information - just a couple of strings basically - binary data

We have a problem with getting binary data. Both sides are written in Java 5 so we have couple of ways....

Web Service is not the best solution because of speed, memory etc...

So, What would you prefer?

I would like to miss low level socket connection if possible...

thanks in advance

Vitek

Answer 1

I've tried converting the binary data to Base64 and then sending it over via SOAP calls and it's worked for me. I don't know if that counts as a web service, but if it does, then you're pretty much stuck with sockets.

Answer 2

You might want to have a look at protobuf, this is the library that google uses to exchange data. Its very efficient and extensible. On a sidenote, Never underestimate the bandwidth of a station wagon full of 1TB harddisks!

Answer 3

Some options:

• You could use RMI which will hide the socket level stuff for you, and perhaps gzip the data...but if the connection fails it won't resume for you. Probably will encounter memory issues too.

• just HTTP the data with a binary mime type (again perhaps configuring gzip on the webserver). similar problem on resume.

• spawn something like wget (I think this can do resume)

• if the client already has the data (a previous version of it), rsync would copy only the changes

Answer 4

I'tried Base64 and SOAP and this approach is good but slow and once I have to transfer big amount of data then it run to OutOfMemmory exception....

And to transfer data using SOAP it has to use StringWriters to do so which is not very efficient.

Actually I'm looking for some library which takes bytes and send them to another point where I found the same data :-)

Yes, RMI looks good....any other hints?

Answer 5

What about the old, affordable and robust FTP? You can for example easily embed an FTP server in your server-side components and then code a FTP client. FTP was born exactly for that (File Transfer Protocol, isn't it?), while SOAP with attachments was not designed with that stuff in mind and can perform very badly. For example you could have a look at:

http://mina.apache.org/ftpserver/

But there are other implementations out there, Apache Mina is just the first one I can recall.

Good luck & regards

Answer 6

I think the only way to do LARGE amounts of data is going to be with raw socket access.

You will hit the Out of Memory issues on large files with most other methods.

Socket handling is really pretty straight forward in Java, and it will let you stream the data without loading the entire file into memory (which is what happens behind the scenes without your own buffering).

Using this strategy I managed to build a system that allowed for the transfer of arbitrarily large files (I was using a 7+ GB DVD image to test the system) without hitting memory issues.

Answer 7

Is sneakernet an option? :P

RMI is well known for its ease-of-use and its memory leaks. Be warned. Depending on just how much data we're talking about, sneakernet and sockets are both good options.

Answer 8

Well we are not talking about such big amount of data....It might be from 20MB to 100MB per one call but this is not possible to do using SOAP ;-)

And I need some authentication...

It is SOA based enviroment so this component is responsible for some kind of data and I have to be carefull who wants them ;-)

Answer 9

Consider GridFTP as your transport layer. See also this question.

Answer 10

Take a look at the W3C standard MTOM to transfer binary data as part of a SOAP service. It is efficient in that it sends as a binary and can also send as buffered chunks. It will also interop with other clients or providers:

How to do MTOM Interop

Server Side - Sending Attachments with SOAP