Memory efficient way of inserting an array of objects with Core Data

Question

I'm working on a piece of code for an iPhone application that fetches a bunch of data from a server and builds objects from it on the client. It ends up creating roughly 40,000 objects. They aren't displayed to the user, I just need to create instances of NSManagedObject and store them to persistent storage.

Am I wrong in thinking that the only way to do this is to create a single object, then save the context? is it best to create the objects all at once, then somehow save them to the context after they're created and stored in some set or array? If so, can one show some example code for how this is done or point me in the direction to code where this is done?

The objects themselves are relatively straight forward models with string or integer attributes and don't contain any complex relationships.

Answer 1

It's probably better to create a single object and save the context.

You have 40k objects. Let's say that creating a single NSManagedObject takes x time units. 40kx time units is probably measurable. While the object creation is happening, the user may quit the app for some reason; users are unpredictable. The next time your app starts, you go through the process all over again. It would not be desirable to create the 39,999th object only to have the user quit the app and lose all that work.

If your app were to create each object and save, you could speed up this process a bit. The app starts up and checks to see if it was able to complete the task the last time it ran. If the task was incomplete, it could try to pick up where it left off.

The single object creation and save method may take a longer time to complete but will have a greater likelihood of completing the task.

In terms of memory consumption, this also minimizes the in memory state of your app. The context isn't tracking 40k objects in memory.

Answer 2

Saving after each object would produce very bad performance. You should have a balance of the saves perhaps every 100 (testing will determine the sweet spot) and then keep track of where you are at in the processing when the user quits.

You get time on exit to store state so you can easily store your position in the data processing (5 blocks of 100 saved) and pick back up where you left off.

Saving every object individually would hammer the disk and slow the app to a crawl.

Answer 3

In any case, don't save after inserting every object, or be prepared for dreadful performances.

Here is the code I use to populate a Core Data repository upon first launch.

#define MAX_UNSAVED_AIRPORTS_BEFORE_SAVE 1000
int numAirports = 0;
int numUnsavedAirports = MAX_UNSAVED_AIRPORTS_BEFORE_SAVE; // *** bug. see below
for (NSDictionary *anAirport in initialAirports) {
    NSAutoreleasePool * pool = [[NSAutoreleasePool alloc] init];

    Airport *newAirport = [NSEntityDescription insertNewObjectForEntityForName:@"Airport" inManagedObjectContext:managedObjectContext];

    newAirport.city         = [anAirport objectForKey:@"city"];
    newAirport.code         = [anAirport objectForKey:@"code"];
    newAirport.name         = [anAirport objectForKey:@"name"];
    newAirport.country_name = [anAirport objectForKey:@"country_name"];
    newAirport.latitude     = [NSNumber numberWithDouble:[[anAirport objectForKey:@"latitude"] doubleValue]];
    newAirport.longitude    = [NSNumber numberWithDouble:[[anAirport objectForKey:@"longitude"] doubleValue]];
    newAirport.altitude     = [NSNumber numberWithDouble:[[anAirport objectForKey:@"altitude"] doubleValue]];

    numAirports++;
    numUnsavedAirports++;
    if (numUnsavedAirports >= MAX_UNSAVED_AIRPORTS_BEFORE_SAVE) {
        if (![managedObjectContext save:&error]) {
            NSLog(@"Unresolved error %@, %@", error, [error userInfo]);
            abort();
        }
        numUnsavedAirports = 0;
    }
    [pool release];
}

Also don't forget to save one last time after the loop.

Also be aware that a bug exists that will lead to a crash if all three of the following conditions are met:

1. The Repository is empty
2. You have a UITableView with sections
3. Your first save saves more than one object.

The workaround in the code above is to initialize the numUnsavedAirports to MAX_UNSAVED_AIRPORTS_BEFORE_SAVE in order to make sure the first save happens after the first insert.

I hope this helps.