.NET Framework Library for arbitrary digit precision
I'm reviving this question, and making it more specific: Is there a .NET framework library that supports numbers with arbitrary digits of precision? ...
numeric-precision
What class to use for money representation?
What class should I use for representation of money to avoid most rounding errors? Should I use Decimal, or a simple built-in number? Is there any existing Money class with support for currency conversion that I could use? Any pitfalls that I should avoid? ...
Why is Math.sqrt(i*i).floor == i?
I am wondering if this is true: When I take the square root of a squared integer, like in f = Math.sqrt(123*123) I will get a floating point number very close to 123. Due to floating point representation precision, this could be something like 122.99999999999999999999 or 123.000000000000000000001. Since floor(122.999999999999999999) ...
Numerical precision of double type in Visual C++ 2008 Express debugger
I'm using Visual C++ 2008 Express Edition and when i debug code: double x = 0.2; I see in debugging tooltip on x 0.20000000000000001 but: typedef numeric_limits< double > double_limit; int a = double_limit::digits10 gives me: a = 15 Why results in debugger are longer than normal c++ precision? What is this strange precision base...
Is the use of machine epsilon appropriate for floating-point equality tests?
This is a follow-up to Testing for floating-point value equality: Is there a standard name for the “precision” constant?. There is a very similar question Double.Epsilon for equality, greater than, less than, less than or equal to, greater than or equal to. It is well known that an equality test for two floating-point values x and y s...
Arithmetic precision with doubles in Matlab
I am having a bit of trouble understanding how the precision of these doubles affects the outcome of arithmetic operations in Matlab. I thought that since both a & b are doubles they would be able to carry out operations up to that precision. I realize there can be round-off error but since these numbers are well within the 64-bit number...