batching remote calls (to database or service)

Question

Hi all,

Hope some of you can give some pointers on this one.

I generate code where i have to make calls to remote resources like webservices or databases.

Consider this piece of code

class Parent{    
    IEnumerable<Child> Children;

    int SumChildren() {  
        // note the AsParallel
        return Children.AsParallel().Sum(c => c.RemoteCall()); 
    }   
}

class Child {        
    public int RemoteCall() {
        // call some webservice. I'd like to pool these calls 
        // without having to rewrite the rest of this code
    } 
}

For 50 children, it's going to make 50 calls to the service, taking the overhead 50 times. In my real life examples this could easily be a million calls, bringing the whole thing to a crawl.

What i would like to do is batch these calls in some way that is transparent for the calling thread/task. So instead of calling the service directly, it calls some central queue (the ' train station') that batches these calls.

So that when it does that, the calling task blocks. Then the queue waits for X calls to accumulate and then makes 1 call to the remote service with a list of requests.

WHen the result comes this queue returns the return values to the right task and unblocks it. for the calling thread, all this remains hidden and it looks like just another function call.

Can this be done? Are there primitives in the TPL that will let me do this?

It kinda smells like the CCR with lots of things going on at the same time waiting for other stuff to complete.

I could of course rewite this code to make the list of requests on the Parent class and then call the service. The thing is that with my real problem all this code is generated. So I would have to 'look inside' the implementation of Child.RemoteCall, making this all a whole lot more complicated than it already is. Also the the Child could be a proxy to a remote object etc. Would be very hard if doable at all, i'd rather isolate this complexity.

Hope this make sense to someone, if not let me know i'll elaborate.

THanks in advance,

John

Answer 1

So that when it does that, the calling task blocks. Then the queue waits for X calls to accumulate

If the queue receives x calls (x < X) then the calling task will block until another task pushes the total >= X. If you only have one task that wants to make N * x calls, it will get stuck.

If your application usually has a lot of tasks running, then you might only see this problem intermittently - where you have unusually low load, or a clean shutdown.

You could solve this by adding a time out, so that the queue will send the batched requests anyway if no requests have been added within a time limit, and/or the first request has been waiting longer than a time limit.

I could of course rewrite this code to make the list of requests on the Parent class and then call the service.

Perhaps you are on the right track with this approach. Could you find a way of replacing the generated method implementation with a hand-coded implementation, by delegation, inheritance, lambda method, or enhancing your generator?

---

... with my real problem all this code is generated.

One point that I'm not quite clear on is which parts of the code are generated (hard to modify) and which parts of the code can be modified to solve this problem?

1. Child.RemoteCall()
2. Parent.SumChildren()
3. Neither of the above.

If it's neither of the above then you have to be able to modify something in order to solve the problem. Are the Parent and Child instances built by an AbstractFactory? If so, then it might be possible to insert a proxy to the Child instances that can be used to modify the non-functional aspects of their behavior(s).

Answer 2

Hi Rich,

(using the answer box for space)

Thanks for the thoughts.

"This would make...": I deal with code that is generated from a repository. When dealing with handcoded example of this problem a developer could spot this and improve it. With generating code it's pretty hard to distill the general case from a set of examples of the problem. This can get pretty complicated so my strategy is devide and conquer. If i have to look inside the function of child that goes out the door.

"I found this link...": I've looked at Futures, but that's more of a forking mechanism that can get parallelized when there's an idle thread.

The TPL seems to be about splitting work up into small bits. What i want to do is take some of these bits and put them together for a while in a different composition and then split them back up again for parallel execution. (I think, still chewing on this one mentally...)

"One more thought": Again, the devide and conquer strategy is what let me get this far. So i devide the larger problem up into little bits, solve them and then put the bits back together. I like to think of an ant colony where each ant follows simple rules (or kanban, that's a similar principle), as opposed to some central management thing (query optimizer) that gets bogged down quickly because it gets very complex very fast.

when the parent could just call the 50 children in a parallelizable fashion, and then these seperate tasks could get batched together just because they're pointing to the same remote resource that would be great.

The major hurdle here is how do i let the calling task (or thread or whatever, a unit of execution) block, have another one pickup the work for a batch of them, do it, put the answer into the collection where all the task put their work and them wake them up again. (and then in an efficient manner..).

I think i remember George Chrysanthakopoulos (The guy that created the CCR) say something that the yield return statement is what he used to to such a thing. I'll try to find that interview again on Channel9.

Regards GJ

Answer 3

You are scratching at the surface of massively parallel programming. You need to think in a concurrency oriented way. You are starting 51 Jobs, and not not the 50 jobs that you need to batch. The extra job is the one that manages the 50 jobs. In terms of the primitives required you need.

JOBHANDLE X= GetJobId();
//single job
AddJob(JOBHANLDE X,ChildJob y);
//container of jobs
AddJobs(JOBHANDLE x, ChildJobs Y);

BeginAsyncExecute(JOBHANDLE X);
WaitTillResult(JOBHANDLE X);

You need an engine in the background that defines blocking primitives (beyond those provided by an OS kernel) and that manages worker threads and jobs to execute, which from the looks of it is handled by the PLINQ technology. PLINQ also uses green threads which is good.

You have mentioned you will have a mix of databases and web-servers. Therefore your Job process/function will have to map the children to the correct resources before the batch is executed. 50 Children might therefore be reduced to far fewer batch-able RPC calls.

So you build up your Job batch and then you block on it.

Getting more specific will be hard. but in light of the discussion so far please tell me what you are having troubles with.