Treating a fat webservice in .net 3.5 c#

Question

I'm dealing with an obese 3rd party webservice that returns about 3mb of data for a simple search results, about 50% of the data in that response is junk.

Would it make sense then to remap this data to my own result object and ditch the response so I'm storing 1-2 mb in memory for filtering and sorting rather than using the web-responses own object and using 2-4 or am I missing a point?

So far I've been accessing the webservice from a separate project and using a new class to provide the interaction and to handle the persistence so my project looks like this

|- Web (mvc2 proj)
|- DAL (database/storage fluent-nhibernate)
|- SVCGateway (interaction layer + webservice related models)
|- Services
--------------
|- Tests
|- Specs

I'm trying to make the application behave fast and I also need to store the result set temporarily in case a customer goes to view the product and wants to go back to the results. (Service returns only 500 of possible 14K results).

So basically I'm looking for confirmation that I'm doing the right thing in pushing the results into my own objects or if I'm breaking some rule or even if there's a better way of handling it.

Thanks

Answer 1

Try and use serialization to store a cut down results object in memory. This will help you immensely. It really is the only way to increase speed of working with results.

I'd also take a look at out of process caching solutions such as NCache (http://www.alachisoft.com/ncache/) and memcached (http://www.infoq.com/news/2007/07/memcached) to increase your speed and scalability further.

Answer 2

Assuming that you will be manipulating the resulting data set multiple times during the lifetime of your application, I think that mapping the web service's result set to your own objects makes a lot of sense.

Aside from the obvious benefit of a decreased memory footprint, you could also design your own internal representation of the data to be optimized for the types of operations that you intend to perform on it. Thus you'd be saving on both processing time and memory usage.

The only case I could see for storing the bloated 3rd party response is if your primary use case involves users looking at the data exactly once. If users rarely came back to their previous searches then you could make a case for just utilizing the original result set.