Actionscript 3 Math inconsistencies

Question

Hello,

I'm trying to build a calculator in Flex / actionscript 3 but have some weird results using the class Math :

trace(1.4 - .4); //should be 1 but it is 0.9999999999999999
trace(1.5 - .5); //should be 1 and it is 1
trace(1.444 - .444); //should be 1 and it is 1
trace(1.555 - .555); //should be 1 but it is 0.9999999999999999

I know there are some issues with floating point numbers, but as you can see, it should at least fail for all of my examples, am I right?

How the problem is solved in other calculators and how should I proceed in order to build a usable calculator in Actionscript 3 please?

Thank you in advance, Adnan

Answer 1

Welcome to IEEE 754 floating point. Enjoy the inaccuracies. Use a fixed-point mechanism if you want to avoid them.

Answer 2

Your results are to be expected, and will be observed in any programming language with a floating point datatype. Computers cannot accurately store all numbers, which causes edge cases like the ones you posted.

Read up on floating point accuracy problems at Wikipedia.

Answer 3

I would assume that most calculators display fewer decimal places than the precision of their floating point. Rounding to fewer decimal places than your level of precision should alleviate this sort of problem, but it won't solve all of the issues.