How to create a custom ADO Multi Dimensional Catalog with no database

Question

Does anyone know of an example of how to dynamically define and build ADO MD (ActiveX Data Objects Multidimensional) catalogs and cube definitions with a set of data other than a database?

Background: we have a huge amount of data in our application that we export to a database and then query using the usual SQL joins, groups, sums etc to produce reports. The data in the application is originally in objects and arrays. The problem is the amount of data is so large the export can take > 2 hours. So I am trying to figure out a good way of querying the objects in memory, either by a custom OLAP algorithm or library, or ADO MD. But I haven't been able to find an example of using ADO MD without a database behind it.

We are using Delphi 2010 so would use ADO ActiveX but I imagine the ADO.NET MD is similar. I realize that if the application data was already stored in a database the problem would solve itself. Also if Delphi had LINQ capability I could query the objects and arrays that way.

Answer 1

The export takes 2 hours? Some companies have dealt with worse! On a nightly basis!

We used to schedule the transfer of data into the warehouse at 3am, and the trigger the cube processing jobs around 6am....then struggle to make 9am availability.

One thing that improved efficiency was making sure that only new data was dealt with, not old values that remained unchanged. For example, our warehouse held restaurant sales for the last 5 years, so there was no need to reload any rows over a month old as they would be the same!

Is it possible to export your entire application's data into a SQL database just once, and then each day after that just add a little bit of new data, or re-export just a subsection of the application? That will help load times.