Adding unique objects to Core Data

Question

I'm working on an iPhone app that gets a number of objects from a database. I'd like to store these using Core Data, but I'm having problems with my relationships.

A Detail contains any number of POIs (points of interest). When I fetch a set of POI's from the server, they contain a detail ID. In order to associate the POI with the Detail (by ID), my process is as follows: Query the ManagedObjectContext for the detailID. If that detail exists, add the poi to it. If it doesn't, create the detail (it has other properties that will be populated lazily).

The problem with this is performance. Performing constant queries to Core Data is slow, to the point where adding a list of 150 POI's takes a minute thanks to the multiple relationships involved.

In my old model, before Core Data (various NSDictionary cache objects) this process was super fast (look up a key in a dictionary, then create it if it doesn't exist)

I have more relationships than just this one, but pretty much every one has to do this check (some are many to many, and they have a real problem).

Does anyone have any suggestions for how I can help this? I could perform fewer queries (by searching for a number of different ID's), but I'm not sure how much this will help.

Some code:

        POI *poi = [NSEntityDescription
    insertNewObjectForEntityForName:@"POI"
    inManagedObjectContext:[(AppDelegate*)[UIApplication sharedApplication].delegate managedObjectContext]];

    poi.POIid = [attributeDict objectForKey:kAttributeID];
 poi.detailId = [attributeDict objectForKey:kAttributeDetailID];
 Detail *detail = [self findDetailForID:poi.POIid];
 if(detail == nil)
 {
  detail = [NSEntityDescription
     insertNewObjectForEntityForName:@"Detail"
     inManagedObjectContext:[(AppDelegate*)[UIApplication sharedApplication].delegate managedObjectContext]];
  detail.title = poi.POIid;
  detail.subtitle = @"";
  detail.detailType = [attributeDict objectForKey:kAttributeType];
 }





-(Detail*)findDetailForID:(NSString*)detailID {
NSManagedObjectContext *moc = [[UIApplication sharedApplication].delegate managedObjectContext];
NSEntityDescription *entityDescription = [NSEntityDescription
            entityForName:@"Detail" inManagedObjectContext:moc];
NSFetchRequest *request = [[[NSFetchRequest alloc] init] autorelease];
[request setEntity:entityDescription];

NSPredicate *predicate = [NSPredicate predicateWithFormat:
        @"detailid == %@", detailID];
[request setPredicate:predicate];
NSLog(@"%@", [predicate description]);

NSError *error;
NSArray *array = [moc executeFetchRequest:request error:&error];
if (array == nil || [array count] != 1)
{
  // Deal with error...
 return nil;
}
return [array objectAtIndex:0];
}

Answer 1

This page provides some help on optimizing performance: http://developer.apple.com/documentation/Cocoa/Conceptual/CoreData/Articles/cdPerformance.html#//apple_ref/doc/uid/TP40003468-SW1

While not very efficient, why not just build them in-memory with a NSDictionary? Read everything from Core Data into a NSDictionary then merge in your data, replacing everything in Core Data.

Answer 2

I've gotten all this to work really well, thanks to Norman, who put me on the write path. I'll post my helper class here for others.

Basically, my helper class will look up if an NSManagedObject exists for some ID, and can create it for some ID. This executes quickly enough for me, with 1,000 find/create operations taking around 2 seconds on my iPhone (I also did a few other things there, pure find/create is likely faster).

It does this by caching a dictionary of all the NSManagedObjects, and checking that cache rather than executing a new NSFetchRequest.

A couple of modifications that could help things speed up even further: 1. Get only selected properties for the NSManagedObjects 2. Only get the identifier property for the NSManagedObject into a dictionary, instead of the whole object. In my performance testing, the single query wasn't the slow part (but with only 1,000 items, I'd expect it to be fast). The slow part was the creation of the items.

  #import "CoreDataUniquer.h"


@implementation CoreDataUniquer

    //the identifying property is the field on the NSManagedObject that will be used to look up our custom identifier
-(id)initWithEntityName:(NSString*)newEntityName andIdentifyingProperty:(NSString*)newIdProp
{
    self = [super init];
    if (self != nil) {
     entityName = [newEntityName retain];
     identifyingProperty = [newIdProp retain];
    }
    return self;
}

-(NSManagedObject*)findObjectForID:(NSString*)identifier
{
    if(identifier == nil)
    {
     return nil;
    }
    if(!objectList)
    { 
     NSManagedObjectContext *moc = [(AppDelegate*)[UIApplication sharedApplication].delegate managedObjectContext];
     NSEntityDescription *entityDescription = [NSEntityDescription
                 entityForName:entityName inManagedObjectContext:moc];
     NSFetchRequest *request = [[[NSFetchRequest alloc] init] autorelease];
     [request setEntity:entityDescription];

     NSError *error;
     NSArray *array = [moc executeFetchRequest:request error:&error];
     objectList = [[NSMutableDictionary dictionary] retain];
     for (NSManagedObject* p in array) {
      NSString* itemId = [p valueForKey:identifyingProperty];
      [objectList setObject:p forKey:itemId];
     }
    }
    NSManagedObject* returnedObject = [objectList objectForKey:identifier];
    return returnedObject;
}
-(NSManagedObject*)createObjectForID:(NSString*)identifier
{

    NSManagedObject* returnedObject = [NSEntityDescription
               insertNewObjectForEntityForName:entityName
               inManagedObjectContext:[(AppDelegate*)[UIApplication sharedApplication].delegate managedObjectContext]];
    [returnedObject setValue:identifier forKey:identifyingProperty];
    [objectList setObject:returnedObject forKey:identifier];
    return returnedObject;
}


- (void) dealloc
{
    DESTROY(entityName);
    DESTROY(identifyingProperty);
    [super dealloc];
}

@end

Answer 3

very helpful QA. thanks;)

Answer 4

You save me a lot of time. Thanks a lot absolut

Answer 5

Check out the section titled "Batch Faulting" on the page titled "Core Data Performance" in Xcode's Core Data Programming Guide that Norman linked to in his answer.

Only fetching those managedObjects whose ids are IN a collection (NSSet, NSArray, NSDictionary) of ids of the objects returned by the server may be even more efficient.

NSSet *oids = [[NSSet alloc] initWithObjects:@"oid1", @"oid2", ..., nil];
NSPredicate *predicate = [NSPredicate predicateWithFormat:@"oid IN %@", oids];
[oids release];

UPDATE: I worked this tip into a solution for the acani usersView. Basically, after downloading a JSON response of users, the iPhone uses the popular open source JSON framework to parse the response into an NSArray of NSDictionary objects, each representing a user. Then, it makes an NSArray of their uids and does a batch fetch on Core Data to see if any of them already exist on the iPhone. If not, it inserts it. If so, it updates the ones that do exist only if their updated attribute is older than that of the one from the server.