Importing large datasets on iPhone using CoreData

Question

Hi there,

I'm facing very annoying problem. My iPhone app is loading it's data from a network server. Data are sent as plist and when parsed, it neeeds to be stored to SQLite db using CoreData.

Issue is that in some cases those datasets are too big (5000+ records) and import takes way too long. More on that, when iPhone tries to suspend the screen, Watchdog kills the app because it's still processing the import and does not respond up to 5 seconds, so import is never finished.

I used all recommended techniques according to article "Efficiently Importing Data" http://developer.apple.com/mac/library/DOCUMENTATION/Cocoa/Conceptual/CoreData/Articles/cdImporting.html and other docs concerning this, but it's still awfully slow.

Solution I'm looking for is to let app suspend, but let import run in behind (better one) or to prevent attempts to suspend the app at all. Or any better idea is welcomed too.

Any tips on how to overcome these issues are highly appreciated! Thanks

Answer 1

Is there any way you can pack the data ahead of time - say during development? And when you push the app to the store, some of the data is already there? That'll cut down on the amount of data you have to pull, thus helping to solve this issue?

If the data is time sensitive, or not ready, or for whatever reason you can't do that, could you compress the data using zlib compression before you ship it over the network?

Or is the problem that the phone dies doing 5K+ inserts?

Answer 2

Instead of pushing plist files to the phone, you might want to send ready to use sqlite files. This has many advantages:

1. no need to import on the phone
2. more compact

If you always replace the whole content simply overwrite the persistent store in the device. Otherwise you may want to maintain an array as plist with all sqlites you have downloaded and then use this to add all stores to the persistentStoreCoordinator.

Bottom line: use several precompiled sqlite files and add them to the persistentStoreCoordinator.

You can use the iPhone Simulator to create those CoreData-SQLite-Stores or use a standalone Mac app. You will need to write both of those yourself.

Answer 3

I imagine you aren't showing all 5K records to the client? I'd recommend doing all of the aggregation you need on the server, and then only sending the necessary data to the phone. Even if this involves generating a few different data views, it'll still be orders of magnitude faster than sending (and then processing) all those rows in the iPhone.

Are you also processing the data in a separate (non event/ui) thread?

Answer 4

I solved a similar problem by putting the insert processing in a background thread. But first I created a progress alert so the user couldn't manipulate the data store while it was inserting the entries.

This is basically the ViewControllers viewDidLoad

- (void)viewDidLoad 
{
    [super viewDidLoad];

    NSError *error = nil;
    if (![[self fetchedResultsController] performFetch:&error]) {
        NSLog(@"Unresolved error %@, %@", error, [error userInfo]);
        abort();
    }

    // Only insert those not imported, here I know it should be 2006 entries
    if ([self tableView:nil numberOfRowsInSection:0] != 2006) {

        // Put up an alert with a progress bar, need to implement
        [self createProgressionAlertWithMessage:@"Initilizing database"];  

        // Spawn the insert thread making the app still "live" so it 
        // won't be killed by the OS
        [NSThread detachNewThreadSelector:@selector(loadInitialDatabase:) 
                                 toTarget:self 
                      withObject:[NSNumber numberWithInt:[self tableView:nil 
                                                numberOfRowsInSection:0]]];
    }
}

The insert thread was done like this

- (void)loadInitialDatabase:(NSNumber*)number
{
    NSAutoreleasePool * pool = [[NSAutoreleasePool alloc] init];

    int done = [number intValue]+1; // How many done so far

    // I load from a textfile (csv) but imagine you should be able to 
    // understand the process and make it work for your data
    NSString *file = [NSString stringWithContentsOfFile:[[NSBundle mainBundle]
                                                pathForResource:@"filename"
                                                         ofType:@"txt"] 
                                               encoding:NSUTF8StringEncoding
                                                  error:nil];

    NSArray *lines = [file componentsSeparatedByString:@"\n"];

    float num = [lines count];
    float i = 0;
    int perc = 0;

    for (NSString *line in lines) {
        i += 1.0;

        if ((int)(i/(num*0.01)) != perc) {
            // This part updates the alert with a progress bar
            // setProgressValue: needs to be implemented 
            [self performSelectorOnMainThread:@selector(setProgressValue:) 
                                   withObject:[NSNumber numberWithFloat:i/num] 
                                waitUntilDone:YES]; 
            perc = (int)(i/(num*0.01));
        }

        if (done < i) // keep track of how much done previously
            [self insertFromLine:line]; // Add to data storage...

    }

    progressView = nil;
    [progressAlert dismissWithClickedButtonIndex:0 animated:YES]; 
    [pool release];
}

It's a bit crude this way, it tries to init the data storage from where it left of if the user happend to stop it the previous times...

Answer 5

First, if you can package the data with the app that would be ideal.

However, assuming you cannot do that then I would do then following:

1. Once the data is downloaded break it into multiple files before import.
2. Import on a background thread, one file at a time.
3. Once a file has been imported and saved, delete the import file.
4. On launch, look for those files waiting to be processed and pick up where you left off.

Ideally sending the data with the app would be far less work but the second solution will work and you can fine-tune the data break up during development.

Answer 6

Any chance you can setup your server side to expose a RESTful web service for processing your data? I had a similar issue and was able to expose my information through a RESTful webservice. There are some libraries on the iphone that make reading from a webservice like that very easy. I chose to request JSON from the service and used the SBJSON library on the iphone to quickly take the results I got and convert them to dictionaries for easy use. I used the ASIHTTP library for making the web requests and queueing up follow up requests and making them run in the background.

The nice thing about REST is that it a built in way for you to grab batches of information so that you don't need to arbitrarily figure out how to break up your files you want to input. You just setup how many records you want to get back, and the next request you skip that many records. I don't know if that is even an option for you, so I'm not going into a lot of code examples right now, but if it is possible, it may be a smooth way to handle it.

Answer 7

Lets accept that Restful (lazy loading) is not an option... I understand you want to replicate. If the load problem is of the type 'less and less rows loading in more and more time) then in psuedo code...

[self sQLdropIndex(OffendingIndexName)]
[self breathInOverIP];
[self breathOutToSQLLite];
[self sQLAddIndex(OffendingIndexName)]

This should tell you lots.