views:

410

answers:

5

I have 100+ channels of video streams to process all at the same time. I need to capture the video, generate thumbnails, and serve them out as a web service. For the generation of thumbnail, I can use JMF etc.(I noticed there is another post talking about how to generate and access: http://stackoverflow.com/questions/772715/better-quality-thumbnails-from-larger-image-files). But my concern is: how to scale? Java EE EJB or simply Java SE Threads? What's the cons and pros? How to scale horizontally using EJB?

I am not that familiar with scalability issue, and I really appreciate your kind suggestions.

Thanks.

A: 

Java SE Threads can help you scale on a single machine, but if you are going to need to scale horizontally across different machines, EJB would be one way to do it.

If it was me, I'd probably farm it out to a separate web service tier that could run on as many machines as needed, and then load balance between those machines.

Spike Williams
Williams, could you explain more on " farm it out to a separate web service tier", thanks.
Lily
Figure out out what the smallest unit of work you would doing that works within a single thread. Create a web service around that unit of work. This service could be run in parallel on a number of different machines, and would have a load balancer in front of it to distribute the work evenly. In front of that, your main web service would have the task of splitting up the work into the single-task subunits, and forwarding those units on the the single-task service via the load balancer. It would collect the results, and send them back to the requesting client.
Spike Williams
+4  A: 

Agree... threads should help to scale on single machine. If you want to scale across different machines - use Terracotta.

eugener
A: 

I don't see a reason to use EJB in this situation. You have to ask yourself where is the bottleneck. My bet will be with the video processing. I would profile your application and see how many threads can be processing before they are spending more time waiting for their timeslice than they are processing. After a point adding more threads will not add more throughput. At that point you know what a machine will do and how many machines you will need to sustain a certain throughput. How you scale across machine is another question.

Javamann
That's a very good point! Do some experiment before buying hardware seems a better solution.
Lily
A: 

These are two distinct problems.

Capturing/processing sounds like a render farm-like problem. Those are scaled trivially horizontally. Most solutions involve a queue of jobs, you don't even need to do this in Java; just find a simple solution you like. "render farm ffmpeg" or stuff like that should yield results in Google.

Your 'serving out as a web service' part is somewhat undefined. If you want those videos to be accesible, you might just need to put them on an HTTP server- those can be load-balanced easily and thus be horizontally scalable- storage speed or network bandwidth would probably be your first bottlenecks there.

alex
A: 

Leave the formal J2EE stack behind.

Rather, a nice message queue that talks JMS with X number of JVMs running Y number of threads as consumers.

Xepoch