I'm having a problem with decimalNumberBySubtracting and decimalNumberByDividingBy.

Question

Simply put, decimalNumberBySubtracting and decimalNumberByDividingBy are not working ... at all. No errors, just messed up results. I really need to fix this but Im not getting any errors and I cant think of any other way for how to do the calculation!! Im getting numbers like:

257849691392950000000000000000000000000000000000000000000000000000000000000000000

Where they should be evalutating to:

-0.581

Here's the long story. I am trying to do a simple Z-Score calculation on a data set. A Zscore is defined as (X - Average)/(stdDev). Pretty simple. My test data includes 5 plates each with 96 samples. This code identifies each individual plate, generates average and stdev for each plate, then for each sample in THAT plate, performs the above calculation.

//identify unique plates
NSSet * uniqueValues = [NSSet setWithArray:[items valueForKey:@"plate"]];
id i;
for (i in uniqueValues) {
    //Lots of NSExpression and Predicate stuff I pulled out cause its working fine

    NSExpression *plateMeanExpression = [NSExpression expressionForFunction:@"average:"
                                                                  arguments:plateArgumentArray];
    NSExpression *plateStdDevExpression = [NSExpression expressionForFunction:@"stddev:"
                                                                    arguments:plateArgumentArray];

    // generate Z Score and StDev:
    NSDecimalNumber *zscoreMean = [plateMeanExpression expressionValueWithObject:nil
                                                                         context:nil];
    NSDecimalNumber *zscoreStdDev = [plateStdDevExpression expressionValueWithObject:nil
                                                                             context:nil];

    //iterate through samples in each plate and perform calculation, update CoreData
    for (ScreenVariables *zValue in eachPlateArray) {
        NSDecimalNumber *deltaTemp = [zValue valueForKey:@"delta"];
        NSDecimalNumber *numerator = [deltaTemp decimalNumberBySubtracting:zscoreMean];
        NSDecimalNumber *zScoreValue = [numerator decimalNumberByDividingBy:zscoreStdDev];
        [zValue setZScore:zScoreValue];

The problem is that when I run this calculation I should get a number from ~(-6,6) with 99% of values falling between -3 and 3. One data point returned is this:

257849691392950000000000000000000000000000000000000000000000000000000000000000000

So OK fine, I did some troubleshooting by inserting the variables deltaTemp, numerator, zscoreMean, zscoreStdDev into my NSTableView. The idea was to see what XCode was evaluating these variables out to and fix whichever is the trouble maker. Did it using this code:

ScreenVariables *addData = [NSEntityDescription insertNewObjectForEntityForName:@"ScreenVariables"
                                                         inManagedObjectContext:self.managedObjectContext];
[addData setValue:deltaTemp forKey:@"someAvailableKey-1"];
[addData setValue:numerator forKey:@"someAvailableKey-2"];
[addData setValue:zscorePlateMean forKey:@"someAvailableKey-3"];
[addData setValue:zscoreStdDev forKey:@"someAvailableKey-4"];

I got values for my variables:

deltaTemp = -919
zscoreMean = -141
numerator = -919    // should be -778.58 = -919 -(-141)
zscoreStdDev = 1337.635067946411

This evaluates to Z-Score= -0.581...

Thats a perfectly valid ZScore ... so the problem is in the:

[deltaTemp decimalNumberBySubtracting:zscoreMean];
[numerator decimalNumberByDividingBy:zscoreStdDev];

The variable numerator isn't evaluating correctly and neither is the zscore value. My CoreData DataModel SPECIFICALLY has each numeric variable's input type defined as NSDecimalNumber so Im assuming my number encoding is fine cause they're getting inserted individually without any problems ...

Answer 1

NSDecimalNumber *zscoreMean = [plateMeanExpression  expressionValueWithObject:nil context:nil];
NSDecimalNumber *zscoreStdDev = [plateStdDevExpression expressionValueWithObject:nil context:nil];

These were evaluating to NSNumber objects, not NSDecimalNumber Objects ... not sure why I didnt get any errors.

    NSNumber *expMean = [plateMeanExpression  expressionValueWithObject:nil context:nil];
    NSNumber *expStdDev = [plateStdDevExpression expressionValueWithObject:nil context:nil];

    NSDecimalNumber *zscoreMean = [NSDecimalNumber decimalNumberWithDecimal:[expMean decimalValue]];
    NSDecimalNumber *zscoreStdDev = [NSDecimalNumber decimalNumberWithDecimal:[expStdDev decimalValue]];

This worked beautifully.