For a single web request, We fetch more than 1000 rows of data from an external system and we need to re-format the data into much better presentation format before we display it to the end user on the web page.
When we loop through the data in ASPX pages, it is creating a hard loop (or tight loop?) which is consuming more than 95% of the CPU.
How do we avoid CPU spikes and also have the process completed as fast as possible? TIA
Are you duplicating any effort here? Is it the same 1000 rows on subsequent requests for the same data? Then maybe you should cache the results? Are some of the calculations the same, over and over? Then maybe you should store the results of the calculations along with the raw data.
Bottom line: don't do a tight loop in the presentation layer. It doesn't belong there.
It's also hurting your scalability, BTW. You're tying up worker threads that could be handling other requests, perhaps other requests not sitting in tight CPU loops.
Building on John's #3, do you really need 1000 rows of data? That sounds like a lot. Users won't appreciate having to sort through that manually. Consider offering paging or search features that reduce the amount shown on the screen at one time. If you're already compiling that down to a smaller number of records, you can probably do it more efficiently in your database.
Unless you're doing a lot of really complicated stuff, I'd wonder how you're noticing your CPU spinning for an noticeable period of time crunching only 1000 rows. You can do some serious work on a modern server in a very short period of time so long as the data is already in memory (which it sounds like it is in your case).
Are you doing things like repeatedly iterating over the items (ie. nested loops) when using a dictionary would work better? Take a close look at what you're doing and consider how many times each line of code actually runs.
Some code I worked on last week had to evaluate a series of complex security rules on about 500,000 objects. The end code took about 3.5 minutes to run -- 2 minutes to load from the DB, 10 seconds to process, and 1 minute 20 seconds to write the results out to the result file. Once you have everyting in memory, if you make sure you don't loop over data more times than you need, things can be surprisingly fast.