I have a program written in C# and some parts are writing in native C/C++. I use doubles to calculate some values and sometimes the result is wrong because of too small precision. After some investigation i figured out that someone is setting the floating-point precision to 24-bits. My code works fine, when i reset the precision to at le...
I want to be able to compare two doubles disregarding a possible precision loss. Is there a method already that handles this case? If not, is there a threshold/guideline to know how much is an adequate equivalence between two doubles? ...
Hello. The following command returns 1: ismember(-0.6, -1:.1:.9) but this next command returns 0: ismember(-0.1, -1:.1:.9) even though -0.1 clearly is in -1:.1:.9. Does anybody know what's going on? ...
Precision loss is one thing, but precision gain??? I have a text file w/ the following coordinates: 41.88694340165634 -87.60841369628906 When I paste this into SQL Server Mgmt Studio table view, it results in this: 41.886943401656339 -87.608413696289062 Am I dreaming? How is this possible? I'm pasting from notepad, and it's r...
I am using TCP/IP socket programming. I have a floating point value stored in a variable ret_val in my server code which I want to send to the client which is waiting to receive it. How can I do it? ...
I have a program in Perl that works with probabilities that can occasionally be very small. Because of rounding error, sometimes one of the probabilities comes out to be zero. I'd like to do a check for the following: use constant TINY_FLOAT => 1e-200; my $prob = calculate_prob(); if ( $prob == 0 ) { $prob = TINY_FLOAT; } This wor...
Hello everyone, Here is a problem that has had me completely baffled for the past few hours... I have an equation hard coded in my program: double s2; s2 = -(0*13)/84+6/42-0/84+24/12+(6*13)/42; Every time i run the program, the computer spits out 3 as the answer, however doing the math by hand, i get 4. Even further, after inputti...
Is it OK to send over network double floating point values (adjusted for correct byte order of course) and using them interchangeably on different cpu architectures, specifically i386, mips (couple of different cores), powerpc (e300, e500). No extremely old hardware. Using gcc 4.2.1 as compiler with -Os for all architectures. Supposedl...
In short: how can I execute a+b such that any loss-of-precision due to truncation is away from zero rather than toward zero? The Long Story I'm computing the sum of a long series of floating point values for the purpose of computing the sample mean and variance of the set. Since Var(X) = E(X2) - E(X)2, it suffices to maintain running ...
Due to the difficulty for machine to represent floating point values exactly, we are using the a technique from Write Great Code: Understanding the machine to perform floating point comparisons: from the editor: please insert your code here. See HTML comment in the question source Currently, we hard coded the 'error' value. But the er...
I have two integer values a and b, but I need their ratio in floating point. I know that a<b and I want to calculate a/b, so if I use integer division I'll always get 0 with a remainder of a. How can I force c to be a floating point number in Python when: c = a / b ...
Is it possible to get the raw bytes of a floating point (IEEE-754) Number object in Actionscript? Or alternately, if I can get the sign (1 bit), mantissa (52 bits) and exponent (11 bits), then I can do the bitshifting myself and construct the raw byte array. I'd like to create precise, compact string representations of Number values (h...
I'm probably completely wrong, and I don't really know anything about it, but I have a question about decimal number data types in programming languages. I understand that floats aren't completely precise, because they're stored in binary with a power or something, but I always wondered why decimal number data types don't just store a nu...
This is very strange to me: irb(main):012:0> "100.7".to_f.modulo(1) => 0.700000000000003 Why the 3 at the end? irb(main):019:0> "10.7".to_f.modulo(1) => 0.699999999999999 Same thing here...we are only getting the remainder of this value divided by one. It should be exact. ...
I am trying to do a SELECT match on a table based upon an identifier and a price, such as: SELECT * FROM `table` WHERE `ident`='ident23' AND `price`='101.31'; The above returns zero rows, while if you remove the price='101.31' bit it returns the correct row. Doing a... SELECT * FROM `table`; Returns the same row as above and qui...
Say I'm inserting a 64 bit floating point double into a DECIMAL(17,5) field. Does the value get rounded or truncated? ...
Is it possible to use RSpec .should(change(...)).by(...) with float numbers and set the compare precision like this: lambda { ...}.should change(unit, :price).by(12.151, 10e-5) Thanks, ...
Background: I receive a long and lat as parameters to a web service. They are typically up to 6 decimal places. When a new request is received, I calculate the distance between the last recorded loc and the long/lat in the params of the request. If the distance is greater than a certain threshold of miles apart, I update the current loc....
I wonder what's the point of NSDecimalNumber. It offers some arithmetics methods, but why should I use NSDecimalNumber and not just double or NSNumber? Did apple take care of some floating point arithmetics uglyness there? Would it make life easier when making heavy use of high precision and big floating point maths? ...
int is usually 32 bits, but in the standard, int is not guaranteed to have a constant width. So if we want a 32 bit int we include stdint.h and use int32_t. Is there an equivalent for this for floats? I realize it's a bit more complicated with floats since they aren't stored in a homogeneous fashion, i.e. sign, exponent, significand. ...