What is the most efficient way to produce a report against a large set of in memory data in .NET?

Question

I am working on a project that sends bills out to hundreds of thousands (if not millions) of customers each month. A lot of the data needs to be manipulated, with interest and whatnot dynamically calculated on the fly through an internal API.

Right now I have the process which works in chunks of 2,500 bills at a time and writes out an XML file containing all of the data when each range completes. I then use Crystal Reports to report on that data, which takes several minutes. I am looking for a better way to report of this data and need some advice.

Is Crystal just slow at parsing XML? I tried throwing Crystal an in-memory data set, but it seems that once I go over 500 records or so, the XML route is actually more efficient. Should I write the data out to our SQL database and report on it that way using index optimizations? Or am I missing an alternate approach that would be much faster?

Answer 1

I ran into the same thing with CR in the past. When I hit that issue, I switched the new development to XML Stylesheets to HTML. It executed faster but took longer to develop.

Answer 2

In my experience, Crystal Reports is just slow on very large reporting projects.

I've found that some other reporting solutions tend to handle large reporting projects better, especially if they're generated off an in-memory dataset. I'd recommend trying Telerik's reporting tool for this.

Answer 3

Personally, I found Crystal Reports to be a memory hog, taking a long time to start up for even small data sets. We also develop in VS 2008 for users using Win2K, so there was a version conflict, as the version of CR that comes with VS 2008 only works on XP or higher.

I wound up converting all of our reports to in-memory objects serialized to XML, which are translated via XSL to HTML/CSS, and can be printed using the WebBrowser control. Now our reports load as soon as the XML is built, rather than waiting for Crystal to start up, which dropped the viewing time from ~30 seconds to under a second, and dropped our MSI size from ~27MB to 1.5MB.

Answer 4

Have you given thought to multi threading the reports? Or using multiple servers/services to handle subsets of the report generation.

We use Crystal reports XI, and have found it very slow. Also, when multi threading from a single service, Crystal reports have a limit to the number of threading reports you can create. We run our reports (smaller than your baches, say about 30 000 reports) using PdfSharp and stored procedures, from Sql Server 2005. And this is a very time consuming process.

So have a look at multiple services/ servers as an option to generate the subsets.

EDIT

as an after thought, i have put out Crystal Question before X-)