The answer to this question may be obvious to someone with more experience in data-warehousing and BI, but I am looking for some guidance.
I'm building a system that uses multiple JMS queues to process millions of messages per day. I need visibility into the activity of these queues, so that I can create reports like..."Yesterday at 11:01am, how many messages entered queue A that had the word 'Foo' in them?"
To make matters worse, I have about 200k words I need to run this report on, for every minute of every day, across several queues, each processing millions of messages per day.
When I think of implementing a custom solution for this, I start going down a wormhole into a pit of despair. Surely I can't be the only person who has ever faced this problem before.
Anybody got any bright ideas?