querying multiple servers

Question

Hi everyone

I am building a web service which needs to query 1 to n file servers, and receive multiple files as a result. Does anyone have a good idea on doing this? Would Threads do any good job? What if the connection with some servers takes longer than the others? How do I know if I really have ALL the queried files?

Thanks

Answer 1

Your question is quite generic and so will be my answer, anyway I hope it will be useful.

I would say you have two options here:

1. Use an asynchronous model. You open the connections to the N file servers, and set up a callback (or an event) that will fire whenever data from one server is received (usually these callbacks will be invoked in a new thread, but check the documentation for your working framework). You get the connection identifier from the data passed to the callback/event, and update the appropriate file.

2. Use a syncrhonous polling model. You open the connections to the N file servers, and enter a loop in which you poll each connection for new data; when new data is available, you update the appropriate file. You exit the loop when all files are completely downloaded.

As per how you know when all files are complete, there is no automatic way for that. You need to establish a convention, known by both you and the server, as to how to know that a file has been completely sent. Options include: the server closes the connection when the file is complete (not very safe as the connection can be accidentally closed), the server sends the file size prior to the file contents, and the end of file is signaled by a special character sequence (this works better on text files where the bytes of the end of file sequence will not normally occur).