Core Data "Upsert" from SQLite Database

Question

I am currently writing an App that needs the ability to modify and persist various pieces of data. I've decided to use Core Data for this purpose. When the user opens the Application for the first time I need to import a large amount of data from a sqlite database, this data consists of the many-to-many relationships.

I'd like to find out the best way to insert all of this data into my code data store. Right now I am using an NSOperation to do the import while that rest of the application remains active, so the user can do other things, but I'd like the import to happen as quickly as possible so the entire App can be accessed right away.

The method that I'm using now is to use an NSFetchRequest to attempt to find the related entity in the data store, if the entity is there I just add it as a relationship, if the entity is not there I create a new one and add it as a relationship. This works, but I feel that it is probably not even close to optimal.

The code I'm using now:

- (void)importEntitiesIntoContext: (NSManagedObjectContext*)managedObjectContext
{
    // Setup the database object
    static NSString* const databaseName = @"DBName.sqlite";
    NSString* databasePath = [[[NSBundle mainBundle] resourcePath] stringByAppendingPathComponent: databaseName];

    sqlite3* database;

    // Open the database from the user's filessytem
    if ( sqlite3_open_v2( [databasePath UTF8String], &database, SQLITE_OPEN_READONLY, NULL ) == SQLITE_OK ) 
    {
        // Setup the SQL Statement
        NSString* sqlStatement = [NSString stringWithFormat: @"SELECT some_columns FROM SomeTable;"];

        sqlite3_stmt* compiledStatement;
        if ( sqlite3_prepare_v2( database, [sqlStatement UTF8String], -1, &compiledStatement, NULL ) == SQLITE_OK ) 
        {
            // Create objects to test for existence of exercises
            NSPredicate* predicate = [NSPredicate predicateWithFormat: @"something == $SOME_NAME"];

            NSEntityDescription* entityDescription = [NSEntityDescription entityForName: @"SomeEntity" 
                                                                 inManagedObjectContext: managedObjectContext];
            NSFetchRequest* fetchRequest = [[[NSFetchRequest alloc] init] autorelease];
            [fetchRequest setEntity: entityDescription];

            // Loop through the results and add them to the feeds array
            while ( sqlite3_step( compiledStatement ) == SQLITE_ROW ) 
            {
                NSString* someName = [NSString stringWithCharsIfNotNull: (char*)sqlite3_column_text( compiledStatement, 1 )];
                NSPredicate* localPredicate = [predicate predicateWithSubstitutionVariables: 
                                               [NSDictionary dictionaryWithObject: someName
                                                                           forKey: @"SOME_NAME"]];
                [fetchRequest setPredicate: localPredicate];

                NSError* fetchError;
                NSArray* array = [managedObjectContext executeFetchRequest: fetchRequest 
                                                                     error: &fetchError];
                if ( array == nil )
                {
                    // handle error
                }
                else if ( [array count] == 0 )
                {
                    SomeEntity* entity = 
                    [NSEntityDescription insertNewObjectForEntityForName: @"SomeEntity"
                                                  inManagedObjectContext: managedObjectContext];
                    entity.name = someName;

// **here I call a method that attempts to add the relationships(listed below)**

                }
                else
                {
                    // Some entity already in store
                }

            }
        }
        else
        {
            NSLog( @"sqlStatement failed: %@", sqlStatement );
        }

        // Release the compiled statement from memory
        sqlite3_finalize( compiledStatement );
    }

    // All the data has been imported into this temporary context, now save it
    NSError *error = nil;
    if ( ![managedObjectContext save: &error] ) 
    {
        NSLog( @"Unable to save %@ - %@", [error localizedDescription] );
    }
}

Method to add the relationships:

- (void)setRelationshipForEntity: (Entity*)entity 
            inManagedObjectContext: (NSManagedObjectContext*)managedObjectContext 
                     usingDatabase: (sqlite3*)database
                        entityId: (NSNumber*)entityId
{

    // Setup the SQL Statement and compile it for faster access
    NSString* sqlStatement = [NSString stringWithFormat: @"SELECT Relationship.name FROM Relationship JOIN Entitys_Relationship ON Entitys_Relationship.id_Relationship = Relationship.id JOIN Entitys ON Entitys_Relationship.id_Entitys = Entitys.id WHERE Entitys.id = %d;", [entityId integerValue]];

    sqlite3_stmt* compiledStatement;
    if ( sqlite3_prepare_v2( database, [sqlStatement UTF8String], -1, &compiledStatement, NULL ) == SQLITE_OK ) 
    {
        // Create objects to test for existence of relationship
        NSPredicate* predicate = [NSPredicate predicateWithFormat: @"relationshipName == $RELATIONSHIP_NAME"];

        NSEntityDescription* entityDescription = [NSEntityDescription entityForName: @"EntityRelationship" 
                                                             inManagedObjectContext: managedObjectContext];
        NSFetchRequest* fetchRequest = [[[NSFetchRequest alloc] init] autorelease];
        [fetchRequest setEntity: entityDescription];

        while ( sqlite3_step( compiledStatement ) == SQLITE_ROW ) 
        {
            NSString* relationshipName = [NSString stringWithCharsIfNotNull: (char*)sqlite3_column_text( compiledStatement, 0 )];
            NSPredicate* localPredicate = [predicate predicateWithSubstitutionVariables: 
                                           [NSDictionary dictionaryWithObject: relationshipName
                                                                       forKey: @"RELATIONSHIP_NAME"]];
            [fetchRequest setPredicate: localPredicate];

            NSError* fetchError;
            NSArray* array = [managedObjectContext executeFetchRequest: fetchRequest 
                                                                 error: &fetchError];
            if ( array == nil )
            {
                // handle error
            }
            else if ( [array count] == 0 )
            {
                EntityRelationship* entityRelationship = 
                [NSEntityDescription insertNewObjectForEntityForName: @"EntityRelationship"
                                              inManagedObjectContext: managedObjectContext];
                entityRelationship.relationshipName = relationshipName;

                [entity addRelationshipObject: entityRelationship];
                //NSLog( @"Inserted relationship named %@", relationshipName );
            }
            else
            {
                [entity addRelationship: [NSSet setWithArray: array]];
            }
        }
    }
    else
    {
        NSLog( @"slqStatement failed: %@", sqlStatement );
    }

    // Release the compiled statement from memory
    sqlite3_finalize( compiledStatement );  
}

Answer 1

Where is the original database coming from?

Typically, you would convert all your data to Core Data during development and then ship the app with a pre-populated Core Data store (no need for a user to wait for an import).

Answer 2

Apple has some suggestions on how to optimize large imports into a Core Data store:

• Disable the undo-manager during batch import
• Don't insert record-by-record - Create batches of size n (depending on record size)
• Use a local autorelease pool and drain it after each batch

See the documentation for the details.