Problem looking at data between 0 and -1...

Question

I'm trying to write a program that cleans data, using Matlab. This program takes in the max and min that the data can be, and throws out data that is less than the min or greater than the max. There looks like a small issue with the cleaning part. This case ONLY happens when the minimum range of the variable being checked is 0. If this is the case, for one reason or another, the program won't throw away data points that are between 0 and -1. I've been trying to fix this for some time now, and noticed that this is the only case where this happens, and if you try to run a SQL query selecting data that is < 0, it will leave out data between 0 and -1, so effectively the same error as what's happening to me. Wondering if anyone might recognize this and know what it could be.

Answer 1

-1 is less than 0, so 0 should be the max value. And if this is the case it will keep points between -1 and 0 by your definition of the cleaning operation:

and throws out data that is less than the min or greater than the max.

If you want to throw away (using the above definition)

data points that are between 0 and -1

then you need to set 0 as the min value and -1 as the max value --- which does not make sense.

Also, I think you mean

and throws out data that is less than the min AND greater than the max.

Answer 2

If I try to mimic your situation with SQL, and run the following query against a datatable that has 1.00, 0.00, -0.20, -0.80. -1.00, -1.20 and -2.00 in the column SomeVal, it correctly returns -0.20 and -0.80, which is as expected.

SELECT  SomeVal
FROM    SomeTable
WHERE   (SomeVal < 0) AND (SomeVal > - 1)

The same is true for MatLab. Perhaps there's an error in your code. Dheck the above statement with your own SELECT statement to see if something's amiss.

Answer 3

I can imagine such a bug if you do something like

minimum = 0
if minimum and value < minimum

Answer 4

It may be that the floats are getting casted to ints before the comparison. I don't know matlab, but in python int(-0.5)==0, which could explain the extra data points getting in. You can test this by setting the min to -1, if you then also get values from -1 to -2 then you'll need to make sure casting isn't being done.

Answer 5

I would write such a function as:

function data = cleanseData(data, limits)
    limits = sort(limits);
    data = data( limits(1) <= data & data <= limits(2) );
end

an example usage:

a = rand(100,1)*10;

b = cleanseData(a, [-2 5]);
c = cleanseData(a, [0 -1]);