Can a web service return a stream?

Question

Hi all,

I've been writing a little application that will let people upload & download files to me. I've added a web service to this applciation to provide the upload/download functionality that way but I'm not too sure on how well my implementation is going to cope with large files.

At the moment the definitions of the upload & download methods look like this (written using Apache CXF):

boolean uploadFile(@WebParam(name = "username") String username,
    @WebParam(name = "password") String password,
    @WebParam(name = "filename") String filename,
    @WebParam(name = "fileContents") byte[] fileContents)
    throws UploadException, LoginException;

byte[] downloadFile(@WebParam(name = "username") String username,
    @WebParam(name = "password") String password,
    @WebParam(name = "filename") String filename) throws DownloadException,
    LoginException;

So the file gets uploaded and downloaded as a byte array. But if I have a file of some stupid size (e.g. 1GB) surely this will try and put all that information into memory and crash my service.

So my question is - is it possible to return some kind of stream instead? I would imagine this isn't going to be terribly OS independent though. Although I know the theory behind web services, the practical side is something that I still need to pick up a bit of information on.

Cheers for any input, Lee

Answer 1

Seeing as web services return objects via serialization, there is definitely no way to create a stream.

However you could just provide chunking.

boolean uploadFile(String username, String password, String fileName, int currentChunk, int totalChunks, byte[] chunk);

Would require some footwork in cases where you don't get the chunks in the right order (Or you can just require the chunks come in the right order), but it would probably be pretty easy to implement.

Answer 2

One way to do it is to add a uploadFileChunk(byte[] chunkData, int size, int offset, int totalSize) method (or something like that) that uploads parts of the file and the servers writes it the to disk.

Answer 3

Yes, it is possible with Metro. See the Large Attachments example, which looks like it does what you want.

JAX-WS RI provides support for sending and receiving large attachments in a streaming fashion.

• Use MTOM and DataHandler in the programming model.
• Cast the DataHandler to StreamingDataHandler and use its methods.
• Make sure you call StreamingDataHandler.close() and also close the StreamingDataHandler.readOnce() stream.
• Enable HTTP chunking on the client-side.

Answer 4

When you use a standardized web service the sender and reciever do rely on the integrity of the XML data send from the one to the other. This means that a web service request and answer only are complete when the last tag was sent. Having this in mind, a web service cannot be treated as a stream.

This is logical because standardized web services do rely on the http-protocol. That one is "stateless", will say it works like "open connection ... send request ... receive data ... close request". The connection will be closed at the end, anyway. So something like streaming is not intended to be used here. Or he layers above http (like web services).

So sorry, but as far as I can see there is no possibility for streaming in web services. Even worse: depending on the implementation/configuration of a web service, byte[] - data may be translated to Base64 and not the CDATA-tag and the request might get even more bloated.

P.S.: Yup, as others wrote, "chuinking" is possible. But this is no streaming as such ;-) - anyway, it may help you.

Answer 5

Keep in mind that a web service request basically boils down to a single HTTP POST.

If you look at the output of a .ASMX file in .NET , it shows you exactly what the POST request and response will look like.

Chunking, as mentioned by @Guvante, is going to be the closest thing to what you want.

I suppose you could implement your own web client code to handle the TCP/IP and stream things into your application, but that would be complex to say the least.

Answer 6

I think using a simple servlet for this task would be a much easier approach, or is there any reason you can not use a servlet?

For instance you could use the Commons open source library.

Answer 7

The RMIIO library for Java provides for handing a RemoteInputStream across RMI - we only needed RMI, though you should be able to adapt the code to work over other types of RMI . This may be of help to you - especially if you can have a small application on the user side. The library was developed with the express purpose of being able to limit the size of the data pushed to the server to avoid exactly the type of situation you describe - effectively a DOS attack by filling up ram or disk.

With the RMIIO library, the server side gets to decide how much data it is willing to pull, where with HTTP PUT and POSTs, the client gets to make that decision, including the rate at which it pushes.

Answer 8

For WCF I think its possible to define a member on a message as stream and set the binding appropriatly - I've seen this work with wcf talking to java web service

you need to set the transferMode="StreamedResponse" in the httpTransport configuration and use mtomMessageEncoding (need to use a custom binding section in the config)

I think one limitation is that you can only have a single message body member if you want to stream (which kind of makes sense)

Answer 9

Apache CXF supports sending and receiving Streams, Please check out the following link.

http://old.nabble.com/sending-receiving-a-file-in-a-JAX-RS-%28JSR-311%29-application-td25814505.html#a25814505