I have a C# List, that is filled from a database.. So far its only 1400 records, but I expect it to grow a LOT.. Routinely I do a check for new data on the entire list.. What I'm trying to figure out is this, is it faster to simply clear the List and reload all the data from the table, or would checking each record be faster..
Intuition tells me that the dump and load method would be faster, but I thought I should check first...
Depends how slow the load is. I'd say access something in memory is always going to be faster than loading from a DB. However you need to do your own calculations.
Remember don't optimise without numbers to back you up.
You should dump and reload, definitely. I base this advice purely on my (perhaps unwarranted) fear of your code that checks for new data.
If my understanding is correct, you would have to load the list from MySql anyway, and veryfy that the in memory list is up to date, correct? So then the only issue you refer to is the in memory manegement of the list.
Well, typically I would try to profile the different behaviours first, an see which performs better.
But as you state, I would think that a clear and recreate should be faster than a systematic check and update.
If the database is not getting hit several times an hour, then definitly reload the data from the database. Offloding as much work to the database as it is designed to take full advantage of system resources.
Add a column called 'InsertedTime' in your database. At intervals update those rows with InsertedTime>ListCreatedTime(a variable in your app)
This assumes that your db data are not deleted
If you want to handle db changes like updation and deletion. You can better set flags in your db. Later when you've updated your list, you can delete your records from your app.
But anyways profile your dump and load method and compare it with this to know which is faster in your scenario.