Downloading large files via web service in Java

Question

I have a web application sitting out there that is a repository for files. This web application provides web services that allow clients to search the repository and download any of the attachments via SOAP.

Currently I have tried to use Spring-WS 1.5.8 with MTOM to send the attachment to the client, but I keep getting out of memory errors. I don't believe these errors are related to my Tomcat 6 instance because my server has 8GB of memory and I configured Tomcat to use 4GB of that. I am getting these errors on files as small as 200MB.

I need to use SOAP, even though it's probably not the best approach at all. I would prefer a solution in Spring, but if that's not possible then I am open to other ideas. I read that one can use the AxiomSoapMessageFactory to stream files to the server for uploading purposes, but not the other way around. Is this true? I am using Java 6.

Here is the error that I keep getting within the Spring WS Framework:

java.lang.OutOfMemoryError: Java heap space
    com.sun.xml.internal.messaging.saaj.util.ByteOutputStream.ensureCapacity(Unknown Source)
    com.sun.xml.internal.messaging.saaj.util.ByteOutputStream.write(Unknown Source)
    com.sun.xml.internal.messaging.saaj.packaging.mime.internet.BMMimeMultipart.find(Unknown Source)
    com.sun.xml.internal.messaging.saaj.packaging.mime.internet.BMMimeMultipart.readBody(Unknown Source)
    com.sun.xml.internal.messaging.saaj.packaging.mime.internet.BMMimeMultipart.getNextPart(Unknown Source)
    com.sun.xml.internal.messaging.saaj.packaging.mime.internet.BMMimeMultipart.parse(Unknown Source)
    com.sun.xml.internal.messaging.saaj.packaging.mime.internet.BMMimeMultipart.parse(Unknown Source)
    com.sun.xml.internal.messaging.saaj.packaging.mime.internet.MimeMultipart.getCount(Unknown Source)
    com.sun.xml.internal.messaging.saaj.soap.MessageImpl.initializeAllAttachments(Unknown Source)
    com.sun.xml.internal.messaging.saaj.soap.MessageImpl.getAttachments(Unknown Source)
    org.springframework.ws.soap.saaj.Saaj13Implementation.getAttachment(Saaj13Implementation.java:305)
    org.springframework.ws.soap.saaj.SaajSoapMessage.getAttachment(SaajSoapMessage.java:226)
    org.springframework.ws.support.MarshallingUtils$MimeMessageContainer.getAttachment(MarshallingUtils.java:109)
    org.springframework.oxm.jaxb.Jaxb2Marshaller$Jaxb2AttachmentUnmarshaller.getAttachmentAsDataHandler(Jaxb2Marshaller.java:532)
    com.sun.xml.internal.bind.v2.runtime.unmarshaller.MTOMDecorator.startElement(Unknown Source)
    com.sun.xml.internal.bind.v2.runtime.unmarshaller.InterningXmlVisitor.startElement(Unknown Source)
    com.sun.xml.internal.bind.v2.runtime.unmarshaller.SAXConnector.startElement(Unknown Source)
    com.sun.xml.internal.bind.unmarshaller.DOMScanner.visit(Unknown Source)
    com.sun.xml.internal.bind.unmarshaller.DOMScanner.visit(Unknown Source)
    com.sun.xml.internal.bind.unmarshaller.DOMScanner.visit(Unknown Source)
    com.sun.xml.internal.bind.unmarshaller.DOMScanner.visit(Unknown Source)
    com.sun.xml.internal.bind.unmarshaller.DOMScanner.visit(Unknown Source)
    com.sun.xml.internal.bind.unmarshaller.DOMScanner.visit(Unknown Source)
    com.sun.xml.internal.bind.unmarshaller.DOMScanner.visit(Unknown Source)
    com.sun.xml.internal.bind.unmarshaller.DOMScanner.scan(Unknown Source)
    com.sun.xml.internal.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal0(Unknown Source)
    com.sun.xml.internal.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal(Unknown Source)
    javax.xml.bind.helpers.AbstractUnmarshallerImpl.unmarshal(Unknown Source)
    org.springframework.oxm.jaxb.Jaxb2Marshaller.unmarshal(Jaxb2Marshaller.java:421)
    org.springframework.ws.support.MarshallingUtils.unmarshal(MarshallingUtils.java:62)
    org.springframework.ws.client.core.WebServiceTemplate$3.extractData(WebServiceTemplate.java:374)
    org.springframework.ws.client.core.WebServiceTemplate.doSendAndReceive(WebServiceTemplate.java:560)

Answer 1

SOAP/XML in Java has always really a lot of overhead and needs a lot of memory. In this specific case it is trying to allocate a (too big) byte[] in memory instead of writing the stream directly to another kind of OutputStream (anything but ByteArrayOutputStream).

Have you considered just forget the SOAP interface thing at all and go back to the basics using java.net.URLConnection and build on that further? This way you can write the InputStream directly to disk using FileOutputStream which is way more efficient than storing the whole thing in memory.

Answer 2

Looks like you are handling the complete file in memory instead of reading it as it is sent to the client.

Can you diverge this to the web server instead if you create a URL which resolves to the actual file to send and leave it to it?

Answer 3

Possible dup: http://stackoverflow.com/questions/1503666/what-is-the-best-way-to-return-big-chunks-of-binary-data-from-a-webservice

Answer 4

It could have something to do with your eden space being to small. The eden space is the part of the heap where new object are allocated and remain until they have survived a GC. The eden space is by not very big. (i don't have a default value but on default setup with 1GB heap it is only 64MB)

Your file will probably be loaded in the eden space. Either there is not 200 MB free space or the byte array is allocated to small and needs to grow. The only way for an array to hrow in Java is by allocating a new and bigger array and do a memcopy. This will The growth from 100MB to 200MB obviously requires 300 MB of total eden heap space.

You could try setting -XX:NewSize=4196M which will allocate 4GB eden heap space.

I have to say, that i don't known i Tomcat is running in some server mode, that uses a diffrent GC/heap strategy.

You can usevisualgc from jvmstat 3.0 (not the distribution bundled with Java 5 and 6) to monitor the heap an determine which heap space is running full.

You also might want to check out: Tuning Garbage Collection with the 5.0 Java[tm] Virtual Machine

If you resolve this issue you will still face poor performance an a not scalable solution. You will probably be better with some sort of direct streaming. It shouldn't be to hard implementing a simple servlet for that purpose.