Hi all,
I'm working an application that periodically fetch data from a web service. The problem is, the web service does not expose a method to fetch data with a range of time. So I have to fetch all the data and drop the out-dated entries. I think this will be a problem when the scale of return data become large. Is there anyway to enhance the performance? Thanks in advance!
Enhancement should happen on the side of the Web Service. You can only call methods that the service exposes, and if the service always returns all the data, then all data will be wrapped into a SOAP enveloppe if you use SOAP, and be transferred over the wire, there's nothing you can do on your end to enhance performance on this to my knowledge.
Also note that you will take 2 performance hits because of this, firstly because the data that needs to be transferred will grow with time, resulting in longer downloads, and secondly you will need to parse more data.
If possible you should have a overloaded method wherein you can pass startDate and endDate.
If this is not possible then your best bet is to find the fastest way to get the data and filter it. If I assume that you get a data table from the WS method , you can write a LINQ query to filter based on the date range columns and then bulk load to you destination.if for example it is MS Sql Server where you loading data , you can do SQLBulkCopy in first step. There must be similar meathods to do this bulk load.
Once the initial Load is done you can load data to your tables by doing proper indexing on the source tables.
The question is very much centric around the volume of data you might get from WS , what is you destination where data will be loaded , what are the most optimized drivers that you can use for dataread/write and if you can modify the WS or not (I presume not from your question). If you let me more about these things I might be able to give to a specific answer to improve the performance of your load.