Designing for High Volumes of Data in an iPhone App

Question

I have an I phone app which upon opening needs to display a table of data containing roughly 25000 records. Each record consists of two fields: TITLE and DESCRIPTION. Both fields are visible in each table cell. The tableview also is fully searchable with a UISearchBar.

When the application starts and this view is loaded a "collection object" grabs all the data from the table and loads it into "item objects". Since I need both fields for searching I really have nothing to hydrate/dehydrate; thus, the full table is in memory. There must be a better way offer the full search capability but drastically reduce the number of objects in memory at once.

Any ideas?

Answer 1

You should try using a delegate for your table and only load items into memory on demand. If only the first ten items are displayed, the OS will only ask you for those.

You should also consider storing your records in a SQLite database. When a search is active, the delegate will operate on the SQL query that is build using the current search term.

Answer 2

It might be a good idea to search and keep in memory 50-200 or so before/after the current screen so you can scroll items as it loads more from the database.

edit; SQL lite is also a great idea as mentioned above.

Answer 3

I guess I should explain further.

I am already using a SQLite database. My problem is that it is taking too long to load 20,000 records into memory and the application seems sluggish.

Also, If I only have 100 records on either side of the current selection how can I be searching ALL records (including the ones not in memory)?

I'll look into the table delegate suggestion.

Answer 4

I'd re-think your app's architecture.

You have to remember, the iPhone is not the same as a desktop app, and people don't use their phones like they do a regular computer. The iPhone is used for minutes at a time, not hours. No one is ever going to want to scroll through 20,000+ entries. To be honest, I'm not sure anyone would want to do that in a desktop app.

Instead, load a limited number of entries, maybe 100 or 200 tops. Then let the user load more (or search for what they need).

Answer 5

You should be using SQL to search rather than loading everything into memory and searching from there. What point is a database if you are using it like a flat file?

Also, a lot of applications have a "load more" entry as the last in their table view which adds more items to the view.

Answer 6

Perhaps I can clarify what the rest of the group is saying.

As you surely know table view must have a datasource that implements at least the two required methods in the <UITableViewDatasource> protocol :

– tableView:cellForRowAtIndexPath:  
– tableView:numberOfRowsInSection:

I suspect that your current implementation of – tableView:cellForRowAtIndexPath: just grabs the two strings out of your global collection object, pops them into a cell (hopefully one that's being reused properly!) and returns it.

What it probably should be doing is retrieving those data from the database on an as needed basis. I'd implement a class (or classes) to mediate between the app and the database and return blocks of, e.g. 100 rows at a time to whichever class is acting as the datasource (SQLite supports the LIMIT and OFFSET keywords), which will hold onto just that limited set.

When – tableView:cellForRowAtIndexPath: is called by the tableView you can check if the data are in the locally stored block (I'd make a property to store the current record offset) and if not, update the local block with a request to the DB for the next (or previous, as the case may be) 100 records.

Presto. A fixed number (100) of records are in memory at any given time, the datasource provides data to the table view as needed. You could get fancier, but that's the general idea.

As for searching, assuming you're doing a free text search on title, just have the database mediator issue something like

 'select id, title, description from yourTable where title like ?'

and bind the ? to whatever the user typed, with wildcards in appropriate places (don't forget to sanitize your input... not that anyone's terribly likely to perform SQL injection on your iPhone app, but, habits and all... :)

Obviously if you're just searching instead of filtering the results, you'll have to adjust all that a little, but hopefully it's a start.

Good luck!