Scenario
Each night we perform a series of calculations on about a million customer contracts. Each contract is related to one fo a group of about ten products, each of which may employ variations on the set of calculation for be performed (although the overall flow is largely the same). All the contracts for a given product will use the same set of calculations on any given night; the distribution is uneven though ranging from a few thousand for the smallest product to a few hundred thousand for the largest.
Over time we will add new similar (but unlikely to be identical) products and existing products may be adjusted by introducing variations to the algorithms employed. (I expect algorithms to be able to be varied parametrically without code modification, by the way).
The core "engine" we use to support this should be applicable both in product development and production. The latter is managed by our Operations people who are very cautious about what they accept and how change is managed generally.
In considering an architecture I'm strongly drawn to the Open-Closed Principle:
Software entities should be open for extension, but closed for modification.
Since we're planning to implement in C#, I'm considering stipulating that functions be defined externally to the core, loaded in text form and compiled at the start of the run for a product under the control of the current version of the product definition. Each of these components would compile to a class that implements an interface (or equivalent) as defined in the compiled element of the application. I'm optimistic that this would be considered a plus from the Operations point-of-view since the scope of regression tests is dramatically reduced.
The Question(s)
Does this make sense? Can anyone offer any sage advice as to potential pitfalls?
Am I re-inventing dependency injection and would I be better-advised to deliver the regular changes by, say, providing a new DLL each time? If so, how much worse does that make our daily product development process?
Or should I take a more Agile approach, build one product end-to-end and add the others in turn, seeing what architecture emerges as I refactor?
If you're still with me, thanks for your patience...