Core Data performance issues

Question

Hi all,

I have an iPhone app that has the needs to avoid inserting duplicate records, at first I thought I could just test each object in my in-memory array, create a fetch request for each object and test if it exists. But this is proving slow on the device, very slow (about 5 seconds, which is not acceptable).

I have been trying to piece together how to create some smart predicate that I could use in order to get this to work efficiently but without much success.

My objects have a NSNumber field that I have set as the "Identity Property" and also non-optional. This field is called sampleTime (again, this is NOT a date, but a NSNumber)

Here is my idea (borrowed from other threads and even some of my own questions of SO):

Obviously doing a fetch per object (around 380 objects) is not going to work for performance, so I was under the impression that I could do most of it in memory and it would be faster. I need to create some predicate that uses the IN clause, then iterate over that fetch result, testing if any one of the objects is inside that result set, if NOT then insert it, if SO then do nothing.

But my implementation is not working:

NSMutableArray *timeStampArray = [[NSMutableArray alloc] init];
for (id emTmp in myListOfObjects)
    [timeStampArray addObject: [NSNumber numberWithInt:[[emTmp sampleTime] timeIntervalSince1970]]];

NSFetchRequest *fetch = [[[NSFetchRequest alloc] init] autorelease];
[fetch setEntity:[NSEntityDescription entityForName:@"ElectricalMeasurementEntity" inManagedObjectContext:context]];
[fetch setPredicate:[NSPredicate predicateWithFormat:@"sampleTime in %@", timeStampArray]];
NSArray *results = [context executeFetchRequest:fetch error:nil];
int resCount = [results count];

But resCount is always zero, and I dont know why...

BTW, myListOfObjects contains business objects which also have a sampleTime property which IS an NSDate type.

EDIT

Ok, update, I got the basics working. The reason why I was not getting any results was because the loop that created the array was using the id type, which when used the way I was using it was not creating NSNumber objects correctly.

Now I do this:

for (id emTmp in myListOfObjects)
{
    Measurement *t = (Measurement*)emTmp;
    NSTimeInterval d = [t.sampleTime timeIntervalSince1970];
    [timeStampArray addObject: [NSNumber numberWithInt:[[t sampleTime] timeIntervalSince1970]]];
}

which creates a nice list which works very well.

However, I then go on to do this:

NSMutableArray *itemsToInsert = [[NSMutableArray alloc] init];
for (id em in myListOfObjects)
{
    BOOL found = NO;
    Measurement *t = (Measurement*)em;
    for (id e in results) //results from Fetch Request which is now populated properly
    {
        MeasurementEntity *entity = (MeasurementEntity*)e;
        if ([entity.sampleTime isEqualToNumber:[NSNumber numberWithInt:[[t sampleTime] timeIntervalSince1970]]])
        {
            found = YES;
            break;
        }
    }
    if (!found)
        [itemsToInsert addObject: t];
}

This loop (for around 850 objects, on the iPhone 3Gs) takes around 10 - 12 seconds, which I can see why (when 850*850 = 722500 loops!). Can I be more efficient about this?

Thanks

Answer 1

This may seem an obvious suggestion, but if [context executeFetchRequest:fetch error:nil] is not returning any results, seems like the first thing to do is check for errors instead of ignoring them (by setting error:nil).

Something like:

NSError *error = nil;
NSArray *results = [context executeFetchRequest:fetch error:&error];
if (results == nil) {   // fetch failed - huh?
    NSLog(@"Fetch error %@, %@", error, [error userInfo]);
}

Answer 2

You need to strip the fetch down such that it will only check the one property. Then do all your comparisons with predicates for speed. Something like this:

NSArray *newData=[NSArray arrayWithObjects:[NSNumber numberWithInt:1],[NSNumber numberWithInt:6],nil];
NSManagedObject *mo;
for (int i=0; i<5; i++) {
    mo=[NSEntityDescription insertNewObjectForEntityForName:@"Test" inManagedObjectContext:self.moc];
    [mo setValue:[NSNumber numberWithInt:i] forKey:@"numAttrib" ];      
}
[self saveContext];
NSFetchRequest *fetch=[[NSFetchRequest alloc] init];
NSEntityDescription *testEntity=[NSEntityDescription entityForName:@"Test" inManagedObjectContext:self.moc];
[fetch setEntity:testEntity]; 
// fetch only the one property you need to test
NSDictionary *propDict=[testEntity propertiesByName];
[fetch setPropertiesToFetch:[NSArray arrayWithObject:[propDict valueForKey:@"numAttrib"]]];
// Return as dictionaries so you don't have the overhead of live objects
[fetch setResultType:NSDictionaryResultType];
// fetch only those existing property values that match the new data
NSPredicate *p=[NSPredicate predicateWithFormat:@"numAttrib in %@",newData];
[fetch setPredicate:p];

NSArray *fetchReturn=[self performFetch:fetch];//<-- my custom boilerplate
// extract the existing values from the dictionaries into an array
NSArray *values=[fetchReturn valueForKey:@"numAttrib"];
// filter out all new data values that already exist in Core Data
p=[NSPredicate predicateWithFormat:@"NOT (SELF in %@)",values];
NSArray *unmatchedValues=[newData filteredArrayUsingPredicate:p];
NSLog(@"unmatcheValues=%@",unmatchedValues); 

... which outputs:

unmatcheValues=(
    6
)

Now you only need to create new managed objects for the values returned. All other new values already exist.