views:

226

answers:

2

I've studied some CQRS sample implementations (Java / .Net) which use event sourcing as the event store and a simple (No)SQL stores as the 'report store'.

Looks all good, but I seem to be missing something in all sample implementations.

How to handle the addition of new report stores / screens, after an application has gone in production? and how to import existing (latest) data from the event store to the new report store?

Ie:

Imagine a basic DDD/CQRS driven CRM application. Every screen (view really) has it's own structured report store (a SQL table). All these views get updated using handlers listening to the domain events (CustomerCreated / CustomerHasMoved, etc).

One feature of the CRM is that it can log phone calls (PhoneCallLogged event). Due to time constraints we only implemented the logging of phone calls in V1 of the CRM (viewing and reporting of who handled which phone call will be implemented in V2)

After a time running in production, we want to implement the 'reporting' of logged phone calls per customer and sales representative.

So we need to add some screens (views) and the supporting report tables (in the report store) and fill it with the data already collected in the Event Store...

That is where I get stuck while looking at the samples I studied. They don't handle the import of existing (history) data from the event store to a (new) report store.

All samples of the EventRepository (DomainRepository) only have a method 'GetById' and 'Add', they don't support getting ALL aggregate roots in once to fill a new report table.

Without this initial data import, the new screens are only updated for newly occurred events. Not for the phone calls already logged (because there was no report listener for the PhoneCallLogged event)

Any suggestions, recommendations ?

Thanks in advance,

Remco

+4  A: 

You re-run the handler on the existing event log (eg you play the old events through the new event handler)

Consider you example ... you have a ton of PhoneCallLoggedEvents in your event log. Take your new Handles and play all the old events through it. It is then like it has always been running and will just continue to process any new events that arrive.

Cheers,

Greg

Greg Young
So basically you'll need to integrate some kind of one-time event runner (report import) in your deployment process?
Remco Ros
And you'll need 'Query' access to the Event Store: store.GetAllEvents<PhoneCallLogged>();This seems counter intuitive because (as i said) All implementations of an EventStore I have seen (both .net / java) don't allow such read access...
Remco Ros
Usually the provided implementations only contain what is needed at runtime, but not what is needed for maintenance.
Think Before Coding
Very clear. So it's all about separation.Still you need some mechanism to determine which events to send trough this 'maintenance' component. You don't want to miss events or handle the same events twice because of the time frame the new handler subscribes for new events and the initial import of historic events.
Remco Ros
Generally what is best is to:Subscribe to events of that type. Query for previous events (there may be multiple here)let subscriptions queue and process previous events one by one. Then eat off subscribed events (must be idempotent so you don't double process)Part of the reason why the repository does not support it is that you would never want to do this *in* the domain. The repository is the contract that the domain has with the event storage.
Greg Young
It will take some time to fully implement a workable production ready, deployable solution. But at least I learned some more about the deeper concepts. Thanks again.
Remco Ros
A: 

The 'EventRepository' only contains these methods because you only need them in production.

When adding a new denormalization for reporting, you can send all event from start to you handler.

You can do this on your development site this way :

  • Load your event log to the dev site
  • Send all events to your denormalization handler
  • Move your new view + handler to your production site
  • Run events that happened inbetween
  • Now you're ready
Think Before Coding
This is not really doable in the way you describe.We don't have access to production data (events).But based on your and Greg's answer, I see now that you'll need some kind of one-time import component (Event Runner) which is run only once when deploying the new handler to production.
Remco Ros
What prevent you from getting the events ? Once the happened, they'll never change, so there is no risk to make a copy.But anyway, you'll need a component (or process) that reads all events and call the handler.
Think Before Coding