tags:

views:

9128

answers:

9

I am building a java server that needs to scale. One of the servlets will be serving images stored in Amazon S3.

Recently under load, I ran out of memory in my VM and it was after I added the code to serve the images so I'm pretty sure that streaming larger servlet responses is causing my troubles.

My question is : is there any best practice in how to code a java servlet to stream a large (>200k) response back to a browser when read from a database or other cloud storage?

I've considered writing the file to a local temp drive and then spawning another thread to handle the streaming so that the tomcat servlet thread can be re-used. This seems like it would be io heavy.

Any thoughts would be appreciated. Thanks.

+11  A: 

Why wouldn't you just point them to the S3 url? Taking an artifact from S3 and then streaming it through your own server to me defeats the purpose of using S3, which is to offload the bandwidth and processing of serving the images to Amazon.

toby
A: 

You have to check two things:

  • Are you closing the stream? Very important
  • Maybe you're giving stream connections "for free". The stream is not large, but many many streams at the same time can steal all your memory. Create a pool so that you cannot have a certain number of streams running at the same time
Marcio Aguiar
A: 

toby is right, you should be pointing straight to S3, if you can. If you cannot, the question is a little vague to give an accurate response: How big is your java heap? How many streams are open concurrently when you run out of memory?
How big is your read write/bufer (8K is good)?
You are reading 8K from the stream, then writing 8k to the output, right? You are not trying to read the whole image from S3, buffer it in memory, then sending the whole thing at once?

If you use 8K buffers, you could have 1000 concurrent streams going in ~8Megs of heap space, so you are definitely doing something wrong....

BTW, I did not pick 8K out of thin air, it is the default size for socket buffers, send more data, say 1Meg, and you will be blocking on the tcp/ip stack holding a large amount of memory.

Tony BenBrahim
+10  A: 

When possible, you should not store the entire contents of a file to be served in memory. Instead, aquire an InputStream for the data, and copy the data to the Servlet OutputStream in pieces. For example:

ServletOutputStream out = response.getOutputStream();
InputStream in = [ code to get source input stream ];
String mimeType = [ code to get mimetype of data to be served ];
byte[] bytes = new byte[FILEBUFFERSIZE];
int bytesRead;

response.setContentType(mimeType);

while ((bytesRead = in.read(bytes)) != -1) {
    out.write(bytes, 0, bytesRead);
}

// do the following in a finally block:
in.close();
out.close();

I do agree with toby, you should instead "point them to the S3 url."

As for the OOM exception, are you sure it has to do with serving the image data? Let's say your JVM has 256MB of "extra" memory to use for serving image data. With Google's help, "256MB / 200KB" = 1310. For 2GB "extra" memory (these days a very reasonable amount) over 10,000 simultaneous clients could be supported. Even so, 1300 simultaneous clients is a pretty large number. Is this the type of load you experienced? If not, you may need to look elsewhere for the cause of the OOM exception.

Edit - Regarding:

In this use case the images can contain sensitive data...

When I read through the S3 documentation a few weeks ago, I noticed that you can generate time-expiring keys that can be attached to S3 URLs. So, you would not have to open up the files on S3 to the public. My understanding of the technique is:

  1. Initial HTML page has download links to your webapp
  2. User clicks on a download link
  3. Your webapp generates an S3 URL that includes a key that expires in, lets say, 5 minutes.
  4. Send an HTTP redirect to the client with the URL from step 3.
  5. The user downloads the file from S3. This works even if the download takes more than 5 minutes - once a download starts it can continue through completion.
John Vasileff
A: 

In addition to what John suggested, you should repeatedly flush the output stream. Depending on your web container, it is possible that it caches parts or even all of your output and flushes it at-once (for example, to calculate the Content-Length header). That would burn quite a bit of memory.

Johannes Passing
+1  A: 

I agree strongly with both toby and John Vasileff--S3 is great for off loading large media objects if you can tolerate the associated issues. (An instance of own app does that for 10-1000MB FLVs and MP4s.) E.g.: No partial requests (byte range header), though. One has to handle that 'manually', occasional down time, etc..

If that is not an option, John's code looks good. I have found that a byte buffer of 2k FILEBUFFERSIZE is the most efficient in microbench marks. Another option might be a shared FileChannel. (FileChannels are thread-safe.)

That said, I'd also add that guessing at what caused an out of memory error is a classic optimization mistake. You would improve your chances of success by working with hard metrics.

  1. Place -XX:+HeapDumpOnOutOfMemoryError into you JVM startup parameters, just in case
  2. take use jmap on the running JVM (jmap -histo <pid>) under load
  3. Analyize the metrics (jmap -histo out put, or have jhat look at your heap dump). It very well may be that your out of memory is coming from somewhere unexpected.

There are of course other tools out there, but jmap & jhat come with Java 5+ 'out of the box'

I've considered writing the file to a local temp drive and then spawning another thread to handle the streaming so that the tomcat servlet thread can be re-used. This seems like it would be io heavy.

Ah, I don't think you can't do that. And even if you could, it sounds dubious. The tomcat thread that is managing the connection needs to in control. If you are experiencing thread starvation then increase the number of available threads in ./conf/server.xml. Again, metrics are the way to detect this--don't just guess.

Question: Are you also running on EC2? What are your tomcat's JVM start up parameters?

Stu Thompson
A: 

Thanks all for such quality responses.

1/ I will try using the S3 url instead, that is what I did when I encountered the problem last time. In that case I was serving flash (swf) files which were static files in a war. In this use case the images can contain sensitive data so I can't change the S3 ACL to public or others could read them. I guess I could temporarily change the ACL to public read for the duration of the read but that's a bit of a hack. I wonder if there's a way to authenticate a browser to S3 without having an AWS account - that would be the only way to do this properly. I'll have the solve this to be secure enough for a business to use it so any ideas are welcome although it might warrant a second question/thread.

2/ The 2k & 8k suggestions are great - I was arbitrarily using 1k

3/ My hang/OOM error last time was probably due to too many threads trying to complete a stream and then running out of threads and memory. I am reading from the S3 io stream and writing direct to the response output stream (i.e. not loading into memory) but I will try the flushing suggestion as well.

4/ I am using EC2 as my server so reading from S3 is super fast - its got to be a response stream issue if anything. If the client has low bandwidth, does the servlet read/write loop get stalled or does it write to some tomcat buffer? I have Apache in front of tomcat (for fast SSL and "budget" load balancing) so I guess it could buffer there too although my OOM was definitely a java hang - apache was fine.

I'm very grateful for your help, thanks - Steve

Steve Buikhuizen
A: 

Thanks John, that temp key solution is perfect.

I'll use that!

Steve Buikhuizen
A: 

If you can structure your files so that the static files are separate and in their own bucket, the fastest performance today can likely be achieved by using the Amazon S3 CDN, CloudFront.

Emil