How do I fix my output for floating-point imprecision?

Question

Hi,

I am doing some float manipulation and end up with the following numbers:

-0.5
-0.4
-0.3000000000000000004
-0.2000000000000000004
-0.1000000000000000003
1.10E-16
0.1
0.2
0.30000000000000000004
0.4
0.5

The algorithm is the following:

var inc:Number = nextMultiple(min, stepSize);
trace(String(inc));

private function nextMultiple(x:Number, y:Number) {
    return Math.ceil(x/y)*y;
}

I understand the fact the float cannot always be represented accurately in a byte. e.g 1/3. I also know my stepsize being 0.1. If I have the stepsize how could I get a proper output?

The strange thing is that its the first time I've encountered this type of problem. Maybe I dont play with float enough.

Thanks,

Answer 1

The limited floating point precision of binary numbers is your problem, as you recognize. One way around this is not to do floating point math. Translate the problem to integers, then translate back for the output.

Answer 2

If you're using a language with a round function, you can use that.

Edit

In response to comments about rounding, here's a sample in c#:

float value = 1.0F;

for (int i = 0; i < 20; i++)
{
    value -= 0.1F;
    Console.WriteLine(Math.Round(value, 1).ToString() + " : " + value.ToString());
}

The results are:

0.9 : 0.9

0.8 : 0.8

0.7 : 0.6999999

0.6 : 0.5999999

(etc)

The rounding does resolve the precision problem. I'm not arguing that it's better than doing integer math and then dividing by 10, just that it works.

Answer 3

With your specific problem, count from -5 to 5 and divide by 10 before actually using the value for something.

Answer 4

A language agnostic solution would be to store your numbers as an integer number of steps, given that you know your step size, instead of as floats.

A non-language agnostic solution would be to find out what your language's implementation of printf is.

printf ("float: %.1f\n", number);

Answer 5

Either use integers instead of a floating point type, or use a floating point type where the "point" is a decimal point (e.g. System.Decimal in .NET).

Answer 6

I did the following,

var digitsNbr:Number = Math.abs(Math.ceil(((Math.log(stepSize) / Math.log(10))) + 1));      
tickTxt.text = String(inc.toPrecision(digitsNbr));

Its not efficient but i dont have many steps.

====== I should just get the nbr of steps as an int and multiply by step ...

Answer 7

If you don't have printf, or if the steps are not just powers of 10 (e.g. if you want to round to the nearest 0.2) then it sounds like you want a quantizer:

q(x,u) = u*floor(x/u + 0.5);

"u" is the step size (0.1 in your case), floor() finds the greatest integer not greater than its input, and the "+ 0.5" is to round to the nearest integer.

So basically, you divide by the step size, round to the nearest integer, and multiply by the step size.

edit: oh, never mind, you're basically doing that anyway & the step where it's multiplying by u is introducing rounding error.

Answer 8

If it is just the output that you are concerned about:

For a pure actionscript solution check out http://code.google.com/p/printf-as3/ - it looks like it will provide you with a printf function.

If you are using Flex you can use the NumberFormatter class, see http://livedocs.adobe.com/flex/3/langref/mx/formatters/NumberFormatter.html

Answer 9

Simply scale the numbers to obtain integers then do maths and scale them back to floats for display:

//this will round to 3 decimal places
var NUM_SCALE = 1000;

function scaleUpNumber(n) {
    return (Math.floor(n * NUM_SCALE));
}
function scaleDnNumber(n) {
    return (n / NUM_SCALE);
}


var answ = scaleUpNumber(2.1) - scaleUpNumber(3.001);
alert(scaleDnNumber(answ)); // displays: -0.901

Change NUM_SCALE to increase/decrease decimap places

|\/|ax

Answer 10

What Every Computer Scientist Should Know About Floating Point Numbers.

Answer 11

This is a bit counter-intuitive, but I tested it and it works (example in AS3):

var inc:Number = nextMultiple(min, stepSize);
trace(String(inc));

private function nextMultiple(x:Number, y:Number) {
    return Math.ceil(x/y)*(y*10)/10;
}

So the only thing I added is multiplying y by 10, then dividing by 10. Not an universal solution but works with your stepSize.

[edit:] The logic here seems to be that you multiply by a big enough number so as for the last decimal digits to "drop off the scale", then divide again to get a rounded number. That said, the example above which uses Math.round() is more readable and better in the sense that the code explicitly says what will happen to the numbers passed in.

Answer 12

Your best bet is to use a Decimal data type if your language supports it. Decimals were added to a number of languages to combat this exact problem.