Ideal way/architecture to deliver large data over Web Services

Question

We are trying to design 6 web services, which will serve another client component. The client component requires data from the web service we are implementing.

Now, the problem is, there is not 1 Web Service we are implementing, there is one Web Service which the client component hits, this initiates a series (5 more) of Web Services which gather data from their respective data stores and finally provide the data back to the original Web Service, which then delivers the data back to the client component.

So, if the requested data becomes huge, then, this will be a serious problem for our internal communication channel.

So, what do you guys suggest? What can be done to avoid overloading of the communication channel between the internal Web Service and at the same time, also delivering the data to the client component.

Update 1

Using 5 WS, where, 1WS does not know about the others, except the next one is a business requirement. Actually, 5 companies "small services" are being integrated.

We use Java and Axis2

Answer 1

We've had a similar problem. Apart from trying to avoid it (eg for internal communication go direct to db instead of web service) you can mitigate it by at least not performing the 5 or so tasks in series. Make new threads to collect them all in parallel and process them at the end to reduce latency (except where they might contend for the same resource and bottle neck).

But before I'd do anything load test it and see if it is even an issue and get some baseline stats so you can see what improvement each change makes. Also sometimes you might be better off tweaking network settings or the actual network rather than trying to optimise the code - but again test and see.

Answer 2

Put all the data on a temporary compressed file and give back the ftp url of the file.

The client fetches the big data chunk uncompress it and reads it. (maybe some authentication mechanism for the ftp server)