In R, what is the difference between these two?
0.9 == 1-0.1 >>> TRUE 0.9 == 1.1-0.2 >>> FALSE ...
floating-accuracy
What is the status of tcl_precision?
I don't use Tcl in my daily work. However, I have a colleague who occasionally interacts with a customer who wishes our tool's extension language worked more like Tcl (!). One topic he brought up was how Tcl let him set how much precision was stored in a double, via a global variable, tcl_precision. I did some web searches, and the docu...
Floating Point Modulo Problem
Hello everyone, I've stumbled onto a very strange bug. Read the comments in the code to see what the bug is exactly, but essentially a variable modulo 1 is returning 1 (but it doesn't equal 1!). I'm assuming there is a display problem where the float is extremely close to one but not exactly. However, it should be moduloing to zero. I c...
How to calculate a value to a certain number of decimal places?
Using numpy or python's standard library, either or. How can I take a value with several decimal places and truncate it to 4 decimal places? I only want to compare floating point numbers to their first 4 decimal points. ...
Can a calculation of floating point differ on different processors? (+passing doubles between C# and C)
I have an application written in C# that invokes some C code as well. The C# code gets some double as an input, performs some calculations on it, pass it to the native layer that perform its own calculations on it, and then passes back to the C# layer. If i run the same exe/dlls on different machines (all of them are x64 by Intel), is...
jQuery and long int ids
I've faced with a next problem: In our database we have objects with ids, like 4040956363970588323. I'm writing some client-wizard on jQuery for interacting with such objects. Client receives base data about objects trough an Ajax request, like: $.ajax({ url: "/api/pages/", type: "get", dataType: "json", ...
Why does (int)(33.46639 * 1000000) return 33466389?
(int)(33.46639 * 1000000) returns 33466389 Why does this happen? ...
Frac function losing precision.
I have a TDateTime variable which is assigned a value at runtime of 40510.416667. When I extract the time to a TTime type variable using the Frac function, it sets it to 0.41666666666. Why has it changed the precision of the value and is there a workround to retain the precision from the original value ie. to set it to 0.416667. ...
Exact textual representation of an IEEE "double"
I need to represent an IEEE 754-1985 double (64-bit) floating point number in a human-readable textual form, with the condition that the textual form can be parsed back into exactly the same (bit-wise) number. Is this possible/practical to do without just printing the raw bytes? If yes, code to do this would be much appreciated. ...
Why is my number being rounded incorrectly?
This feels like the kind of code that only fails in-situ, but I will attempt to adapt it into a code snippet that represents what I'm seeing. float f = myFloat * myConstInt; /* Where myFloat==13.45, and myConstInt==20 */ int i = (int)f; int i2 = (int)(myFloat * myConstInt); After stepping through the code, i==269, and i2==268. What's ...
c++ floating point precision loss: 3015/0.00025298219406977296
The problem. Microsoft Visual C++ 2005 compiler, 32bit windows xp sp3, amd 64 x2 cpu. Code: double a = 3015.0; double b = 0.00025298219406977296; //*((unsigned __int64*)(&a)) == 0x40a78e0000000000 //*((unsigned __int64*)(&b)) == 0x3f30945640000000 double f = a/b;//3015/0.00025298219406977296; the result of calculation (i.e. "f"...
What Determines the Default Setting of the x87 FPU Control Word?
What determines the default setting of the x87 FPU control word -- specifically, the precision control field? Does the compiler set it based on the target processor? Is there a compiler option to change it? Using Microsoft Visual C++ 2008 Express Edition on an Intel Core Duo processor, the default setting for the precision control field...
Floating point precision in Visual C++
HI, I am trying to use the robust predicates for computational geometry from Jonathan Richard Shewchuk. I am not a programmer, so I am not even sure of what I am saying, I may be doing some basic mistake. The point is the predicates should allow for precise aritmthetic with adaptive floating point precision. On my computer: Asus p...
Fixed Point to Floating Point and Backwards
Is converting Fixed Pt. (fixed n bit for fraction) to IEEE double safe ? ie: does IEEE double format can represent all numbers a fixed point can represent ? The test: a number goes to floating pt format then back to it's original fixed pt format. ...
erroneous Visual C float / double conversion?
In Visual C++ i wrote the following sample in a C++ program: float f1 = 42.48f; double d1 = 42.48; double d2 = f1; I compiled the program with Visual Studio 2005. In the debugger i see the following values: f1 42.480000 float d1 42.479999999999997 double d2 42.479999542236328 double d1 by my knowledege is OK, but d2 is wron...
Is there a floating point value of x, for which x-x == 0 is false?
In most cases, I understand that a floating point comparison test should be implemented using over a range of values (abs(x-y) < epsilon), but does self subtraction imply that the result will be zero? // can the assertion be triggered? float x = //?; assert( x-x == 0 ) My guess is that nan/inf might be special cases, but I'm more int...
Does "epsilon" really guarantees anything in floating-point computations?!
To make the problem short let's say I want to compute expression: a / (b - c) on float's. To make sure the result is meaningful, I can check if 'b' and 'c' are inequal: float EPS = std::numeric_limits<float>::epsilon(); if ((b - c) > EPS || (c - b) > EPS) { return a / (b - c); } but my tests show it is not enough to guarantee eit...
Can floating-point precision be thread-dependent?
I have a small 3D vector class in C# 3.0 based on struct that uses double as basic unit. An example: One vector's y-value is -20.0 straight I subtract a vector with an y-value of 10.094999999999965 The value for y I would expect is -30.094999999999963 (1) Instead I get -30.094999313354492 (2) When I'm doing ...
Manipulating and comparing floating points in java
In Java the floating point arithmetic is not represented precisely. For example following snippet of code float a = 1.2; float b= 3.0; float c = a * b; if(c == 3.6){ System.out.println("c is 3.6"); } else { System.out.println("c is not 3.6"); ...
C++ floating point precision
Possible Duplicate: Floating point inaccuracy examples double a = 0.3; std::cout.precision(20); std::cout << a << std::endl; result: 0.2999999999999999889 double a, b; a = 0.3; b = 0; for (char i = 1; i <= 50; i++) { b = b + a; }; std::cout.precision(20); std::cout << b << std::endl; result: 15.000000000000014211 So.. ...