I need recommendations, design input, references to similar systems or anything that will help in building the system below.
I have been assigned to design a message processing based system. The over all architecture is as follows.
We have
i) Client
ii) Server
iii) Data Collectors - Services that gather data from the internet
There can be n number of Clients, n number of Servers and n number of Data Collectors.
Data Collectors run all the time and they gather relevant data from the internet. Each data collectors is registered with the one or more Server. Being registered here means that only the registered servers can query the data collectors. Then we have clients, each client can access a server to query for a specific data. The server, if required, will break down the query and reroute the query to the concerned data collector(s). And when the server gets the the responses, it will send it to the client who asked for the data.
Main issues are as follows.
i) The system should be asynchronous
ii) The data should be made available to the client as soon as possible. i.e. The data collectors when trying to retrieve the data will start sending the data as soon as they find something, they wont wait for the processing of the query to complete.
iii) The system should be able to handle incomplete message requests or data responses. i.e. the request or response may be too large and we may to break it into parts before sending it to the other end.
iv) The system should be able to handle disconnections between Server, Data collectors and clients.
v) It isnt necessary that all the requests will originate from the client. Server and data collectors may be independently exchanging information, like status messages, progress etc whenever required.
vi) System should scale well
I have an initial design that I plan to put forward as an answer but I want your input first.