I don't use Tcl in my daily work. However, I have a colleague who occasionally interacts with a customer who wishes our tool's extension language worked more like Tcl (!). One topic he brought up was how Tcl let him set how much precision was stored in a double, via a global variable, tcl_precision. I did some web searches, and the docu...
I asked a question earlier about how to deal with rounding issues with floating point numbers in PHP, and was pointed to the bc and gmp libraries. I've looked at the functions in these libraries but nothing jumped out at me when I was looking for one to round off the number. How do you accurately round using these libraries? ...
In C++ you can use std::numeric_limits<FloatingPointType>::digits10 to find out exactly how many places of precision you can get before the decimal place. How do I find out how many decimal places I can get following it, if I know the number will always be fixed-point between 0 and 1 inclusive? Edit: The reason I ask is that I have an...
So, I have a file that looks like this: # 3e98.mtz MR_AUTO with model 200la_.pdb SPACegroup HALL P 2yb #P 1 21 1 SOLU SET RFZ=3.0 TFZ=4.7 PAK=0 LLG=30 SOLU 6DIM ENSE 200la_ EULER 321.997 124.066 234.744 FRAC -0.14681 0.50245 -0.05722 SOLU SET RFZ=3.3 TFZ=4.2 PAK=0 LLG=30 SOLU 6DIM ENSE 200la_ EULER 329.492 34.325 209.775 FRAC 0.70297 0....
Does anybody know how to go out solving this problem? * a = 1.0 × 2^9 * b = −1.0 × 2^9 * c = 1.0 × 2^1 Using the floating-point (the representation uses a 14-bit format, 5 bits for the exponent with a bias of 16, a normalized mantissa of 8 bits, and a single sign bit for the number), perform the following two calculations, paying clo...
How come ceil() rounds up an even floating with no fractional parts? When I try to do this: double x = 2.22; x *= 100; //which becomes 222.00... printf("%lf", ceil(x)); //prints 223.00... (?) But when I change the value of 2.22 to 2.21 x *= 100; //which becomes 221.00... printf("%lf", ceil(x)); //prints 221.00... as expec...
I couldn't find anything that rejects or confirms whether SQL Server 'MONEY' data type is a decimal floating point or binary floating point. In the description it says that MONEY type range is from -2^63 to 2^63 - 1 so this kind of implies that it should be a binary floating point. But on this page it lists MONEY as "exact" numeric. Wh...
Hello everyone, I've stumbled onto a very strange bug. Read the comments in the code to see what the bug is exactly, but essentially a variable modulo 1 is returning 1 (but it doesn't equal 1!). I'm assuming there is a display problem where the float is extremely close to one but not exactly. However, it should be moduloing to zero. I c...
I have an application written in C# that invokes some C code as well. The C# code gets some double as an input, performs some calculations on it, pass it to the native layer that perform its own calculations on it, and then passes back to the C# layer. If i run the same exe/dlls on different machines (all of them are x64 by Intel), is...
I get output files from very old Fortran programs, which look like: 0.81667E+00 -0.12650E+01 -0.69389E-03 0.94381E+00 -0.11985E+01 -0.11502E+00 0.96064E+00 -0.11333E+01 -0.17616E+00 0.10202E+01 -0.12435E+01 -0.93917E-01 0.10026E+01 -0.10904E+01 -0.15108E+00 0.90516E+00 -0.11030E+01 -0.19139E+00 0.98624E+00 -0.11598E+...
I've just started to work with MPFR arbitrary precision library and quite soon encounter very wierd behaviour. The main goal of using it was to improve precision of lagre argument trigs and this works extremly good in MPFR. But then I decided to check out simple math and it was unbeleavable - even in straightforward examples with strict...
Hello, I do some numerical computing, and I have often had problems with floating points computations when using GCC. For my current purpose, I don't care too much about the real precision of the results, but I want this firm property: no matter WHERE the SAME code is in my program, when it is run on the SAME inputs, I want it to give ...
I'm writing a 2D game using OpenGL. When I want to blit part of a texture as a sprite I use glTexCoord2f(u, v) to specify the UV co-ordinates, with u and v calculated like this: GLfloat u = (GLfloat)xpos_in_texture/(GLfloat)width_of_texture; GLfloat v = (GLfloat)ypos_in_texture/(GLfloat)height_of_texture; This works perfectly most of ...
I've faced with a next problem: In our database we have objects with ids, like 4040956363970588323. I'm writing some client-wizard on jQuery for interacting with such objects. Client receives base data about objects trough an Ajax request, like: $.ajax({ url: "/api/pages/", type: "get", dataType: "json", ...
I wish to round a floating point number to set precision and return the result from a function. For example, I currently have the following function: inline bool R3Point:: operator==(const R3Point& point) const { // Return whether point is equal return ((v[0] == point.v[0]) && (v[1] == point.v[1]) && (v[2] == point.v[2])); } What ...
This problem is killing me. How does one roundup a number UP in Python? I tried round(number) but it round the number down. Example: round(2.3) = 2.0 and not 3, what I would like The I tried int(number + .5) but it round the number down again! Example: int(2.3 + .5) = 2 Then I tried round(number + .5) but it won't work in edge cas...
We are stuck with a database that (unfortunately) uses floats instead of decimal values. This makes rounding a bit difficult. Consider the following example (SQL Server T-SQL): SELECT ROUND(6.925e0, 2) --> returns 6.92 ROUND does round half up, but since floating point numbers cannot accurately represent decimal numbers, the "wrong"...
When I run the exact same code performing the exact same floating point calculation (using doubles) compiled on Windows and Solaris, I get slightly different results. I am aware that the results are not exact due to rounding errors. However I would have expected the rounding errors to be platform-independent, thereby giving be the same ...
I know we can't compare 2 floating point values using ==. We can only compare they are within some interval of each other. I know if(val == 0.512) is wrong due to errors inherent in floating point calculations and conversion to binary and should be if (val in (0.512-epsilon, 0.512+epsilon)) But is 0 special? Can we compare floats e...
I can't for the life of me figure out why the following produces the result it does. use POSIX; my $g = 6.65; my $t = $g * 4; my $r = $t - $g; my $n = $r / $g; my $c = ceil($n); print "$c ($n)\n"; Sigil-tastic, I know — sorry. I've solved this for my app as follows: use POSIX; my $g = 6.65; my $t = $g * 4; my $r = $t - $g; my $n = $...