Simple calculation problem with Flex

Question

Im having a trouble with the flex calculation, im trying to transform a pixel value to mm and reversing it.

Starting out with 69.8:

Calculating MM to pixel from: 69.8 mm
69.8*300 = 20940
20940 / 2.54 = 8244.094488188975
8244.094488188975 / 10 = 824.4094488188975

And calculating back:

Calculating pixel to MM from: 824.4094488188975
824.4094488188975/300 = 2.7480314960629917
2.7480314960629917 * 2.54 = 6.979999999999999
6.979999999999999 * 10 = 69.79999999999998

We wanted 69.8 but ended up with 69.79999999999998. I tracked the proccess using simple windows calc and the first place it wents wrong is at 20940 / 2.54 = 8244.094488188975 wich should be 8244,094488188976.

Anny help on this would be great.

Answer 1

Ok, this is not as much a Flex question as it is about general programming. Have you ever wondered how exactly are numbers stored in the computer? The details are more in the real of mathematics, but I'll try to put it simple. You know that there is an infinite number of different Real numbers (eg. like points on a continuous line), but in your computer you store everything as zeroes and ones, and a limited amount of them (32 or 64 "bits"). So now we have a problem - how to represent an unlimited amount of numbers in limited space. The idea used in the floating-point number (ones that can have values withe a "dot" like 1.03 or 4.2232) is that, because you can't have them all, you round the number to the closest one you have.

It's a bit like remembering how much sugar someone puts in his coffee, you don't remember that he likes to have 1.1232 table spoons of sugar, because the table spoon is just not that good in measuring the exact amount of stuff on it. You round it to one and it works fine most of the time.

So with floating point numbers a similar idea stands, with an extra twist - the numbers are much more dense near 0 than away from it, where the "empty space" between them get's really big. For example if you subtract 10000 from the maximal value a number can get it will still be the same, because there is no number as close to it that it makes a difference when looking for the closest one.

    trace (Number.MAX_VALUE == Number.MAX_VALUE-10000);
    //  returns "true"
    trace (200000 == 200000 - 10000);
    //  returns "false"

So your problem comes from assuming that numbers are totally precise while they aren't, you always get things rounded. The Number in as3 adheres to the double-precision IEEE-754 standard, and that's how it decides which number it has and which it rounds.

Look at this:

trace (8244.094488188976);
// returns "8244.094488188975"

Further reading:

Floating point

IEEE_754-2008

Answer 2

Adobe's response in a TechNote entitled Flash returns erroneous results for certain mathematical calculations puts it succinctly:

This is a fact of life in applications that use floating point arithmetic.