Pipeline pattern for message processing

Question

I have a set of services (non-WCF) that sit on queues. When message arrives, typical service does some calculations and spits out zero or more result messages to its output queue. Besides its primary function, each service has some housekeeping logic like auth / audit / logging / status tracking with exact steps and sequencing varying somewhat between the services. This is where the notion of pipeline enters the picture.

I'm not happy with the design we ended up with, and looking for ways to simplify it. Should I model after CCR ports? ASP.NET pipeline? AOP? Anything else?

My current design is as follows: I have an IMessageHandler<TMessage> interface and about 15 implementations that are chained together in 6 unique ways using IoC. The interface defines single method, Handle(TMessage msg), so each implementation can do some logic both before and after passing the message over to the next handler in the daisy chain.

The problem with this is: it's kind of hard to remember what exactly each implementation does and why they were chained this particular way for this particular service. On the other hand, having each aspect sit in its own class allows for better separation of concerns and therefore easier unit-testing.

Ideas? Any good pipeline patterns I can look at? Any nice pipeline implementations I can use as reference? Or should I JFHCI?

Answer 1

I use the design you've chosen quite frequently, not always with IoC.

It sounds like you have a "documentation" problem. The code doesn't do a good job of explaining what it does. I can see a couple of solutions to this problem.

First, you can heavily comment your code. This is often-times a smell and depends a lot on the implementation. You can also comment-in-place by naming your IoC configurations in a particular way.

Second, you can create clumps of objects. For instance, if there are three objects that tend to get run together you can create a "AuthorizeAndAudit" class that delegates to the other objects (probably built using IoC and "plugin families" or whatever your container calls them, if supported). This better ties together the intention of the objects. I tend to have a Collection of IMessageHandler that also implements IMessageHandler and does a foreach on itself.

Third, you can separate out the interfaces. It sounds like you might have a situation where you are splitting objects only because they contain actions that occur in different parts of the chain. You can create one interface (or method on a shared interface) for Authentication, one for Auditing, etc. Your objects can implement on or more of these interfaces. Since the order of the interfaces is defined (call Auth then Audit, etc) your objects can tackle multiple steps in the chain (you end up with a chain of chains) without needing to be split into separate classes. You might even have an object, like a log tracer, that sits on all chains and logs as every step is called.

Beyond that you start getting into something more complex, like workflow, and Windows Workflow might be a good place to look.

Answer 2

I often write code that is similar to what you describe. To make the intent of the code clear, I add a layer of "syntactic sugar" that composes the objects that will do the application's work at runtime, and also clearly expresses what the final object graph does.

Steve Freeman and I wrote a paper about the techniques we use in Java to build this layer of syntactic sugar: http://www.jmock.org/oopsla2006.pdf. I've used the same techniques in C# and other Algol-family languages and built quite complicated "Domain Specific Embedded Languages", including embedded languages for describing financial market models, distributed system architectures, communication protocol stacks, sprite behaviours in games, and so on.