What's the efficient way to search and filter ObservableCollection

Question

I've a database consisting of about 1 million records. My application makes frequent incremental search on these records and filters the UI list. The scenario is more like the 'phone contact search'.
Currently i'm following this:

1. Load the data into List<myModel> in the DataSource Layer`
2. Send it to MainViewModel
3. Load List<myModel> into ObservableCollection<myViewModel> (bound to ListView)
4. According to the keyword, filter the `ObservableCollection
5. While the application is being closed, update the database [Since the ObservableCollection may also be updated by the user]

---

My Questions:

• Is this the efficient way?
  
• Now my applicatation consumes around 30 to 50 MB of memory. Is that fair?
  
• Or should i perform my search in the database instead? [but i cannot compromise on speed]
  
• Should i always create a list of myModel and load it into my ObservableCollection?
  
• Also advise me on the filtering technique that is much suited for incremental search(4th point). Currently I have a GenericList behind containing the entire collection for lookup and add the filtered items to the ObservableCollection, clearing all the previous items.

Edit: Previously i checked the memory consumption with only 37k records. With 250k records, the memory consumption is more than 100 MB:(. Hence now I've planned to keep only some 10k records in memory, If it goes beyond that I'll query the db. Any suggestions?

Thanks in advance, Veer

Answer 1

There are several things you can do without intterrupting the workflow of the user or taking a huge performance hit.

Now my applicatation consumes around 30 to 50 MB of memory. Is that fair?

That's normal for a .net application. You will notice the memory footprint directly after launching the application isn't much smaller.

Or should i perform my search in the database instead? [but i cannot compromise on speed]

Querying the database repeatedly is only appropriate if you couldn't load the data in one step in the first place. Whenever the user types into the box, you want to make sure you aren't querying the database on each change to the search criteria but rather have a short timer in there, that waits for a second or so before querying the database with the new criteria. When requerying the database, it may also help to reduce the number of records shown via paging or lazy loading the rest of the data when the user starts scrolling. Proper database indexing will help you reduce the execution speed of such a query significantly.

If you can however keep the whole list in memory (say it's not too large to query it from the database in one step, or you need to show the whole list to the user anyway), it would be best to keep a pristine copy of the list and have a copy of that list that you can incrementally filter. Therefore you need to check if your search criteria is a subset of the previous search criteria. If so, you can filter the already-filtered list, else you need to filter the pristine list. This can be efficiently implemented using the LINQ .Where() operator, and if necessary parralelized using PLINQ. .Where() exhibits O(n) time AFAIK.

This can be improved on by using a HashSet with appropriate key.