I have just gone through an unexpectedly convoluted process to define datetimes. The underlying data has a better precision than milliseconds. I ended up constructing an intermediate datetime to the nearest second, reading it's value in ticks (10 000 to the millisecond), adjusting the ticks then creating the datetime that I actually wa...
Using the following code snippet: (fromIntegral 100)/10.00 Using the Haskell '98 standard prelude, how do I represent the result with two decimals? Thanks. ...
Why doesn't the average of the score of an employee of each month, when summed, equal the average of the employees score (ever)? Average SELECT Avg(r.score) AS rawScore FROM (ET INNER JOIN Employee AS e ON ET.employeeId = e.id) INNER JOIN (Employee AS a INNER JOIN Review AS r ON a.id = r.employeeId) ON ET.id = r.ETId WHERE (((e.id)=@em...
When I type this into the Visual Studio 2008 immediate window: ? .9 - .8999999999999995 It gives me this as the answer: 0.00000000000000055511151231257827 The documentation says that a double has 15-16 digits of precision, but it's giving me a result with 32 digits of precision. Where is all that extra precision coming from? ...
Hi every one! I am programming on a project which I should store the key of the user to the initial configuration of a machine, I want to write it in C#. I have an initial configuration which consists of two number R and X0, R = 3.9988 and X0 = 0.5. I want to add the user key to these numbers. for example: Key: hos110 => R = 3.9988...
When I construct a boost::options_description instance like options.add_options() ("double_val", value(&config.my_double)->default_value(0.2), "it's a double"); and later want to have the automated output of the options that are available for my program, and put std::cout << options << std::endl; the default value 0.2 is shown wi...
Hi everyone, I've been struggling with precision nightmare in Java and MSSQL up to the point when I don't know anymore. Personally, I understand the issue and the underlying reason for it, but explaining that to the client half way across the globe is something unfeasible (at least for me). The situation is this. I have two columns in ...
Hi, I am entering '35444650.00' as a float into my MySQL and it keeps reformatting to 35444648.00, any help welcome... ...
Hello, Today's problem is as follows: we have a table in an Oracle database. The table contains a field that is of the type Number(18, 3). On the surface, both saving and loading data from said field work perfectly. However, further inspection reveals, that numbers that have three decimal digits in them, e.g. 500.001, are read from the...
Is it possible to create an image or graphic using vector data that has decimal components that vary? And, of course, if its possible... then how? For example: a vector has two points where point one is {9.56, 4.1} and point two is {3.456789,2.12345}. Note: the precision varies from number to number. ...
This is more of a numerical analysis rather than programming question, but I suppose some of you will be able to answer it. In the sum two floats, is there any precision lost? Why? In the sum of a float and a integer, is there any precision lost? Why? Thanks. ...
float totalAmount = 0; . . .//totalAmount assigned value 1.05 correctly . totalAmount += float.Parse(dataRow["Amt"].ToString()); //where dataRow["Amt"] has value 4.93 The answer I get for totalAmount is 5.97999954 instead of 5.98 Why is this happening? ...
Is it possible to set the display precision of a float in Ruby? Something like: z = 1/3 z.to_s #=> 0.33333333333333 z.to_s(3) #=> 0.333 z.to_s(5) #=> 0.33333 Or do I have to override the to_s method of Float? ...
I'm trying to construct an algorithm that validates that a double value is a member of a range defined with min, max and step values. The problem is checking that the value obeys the step rule. For integers this can be easily done: boolean validate(int value, int min, int step, int max){ return value >= min && value ...
This is more of a question of curiosity but does anyone know how negative precision values are handled in C++? For example: double pi = 3.14159265; cout.precision(-10); cout.setf(ios::fixed, ios::floatfield); cout << pi << endl; I've tried this out and using GCC and it seems that the precision value is ignored but I was curious if ...
I have a method that returns a float like 1.234567890.I want to test that it really does so. However, it seems that this returned float has different precision on different platforms so how do I assert that the returned value is 1.23456789? If I just do: $this->assertEqual(1.23456789, $float); Then that might fail on some platforms wh...
Why is the precision of my location in Mapkit not as good as in the maps app? The accuracy is set to best. If i'm trying to get my location in maps.app i have an acc of 50m in my app i habe 500m. What am i doing wron :)? Any tipps to get a better result? Thanks for your help :) ...
In mathematics the identity (1 + sqrt(2))^2 = 3 + 2*sqrt(2) holds true. But in floating point (IEEE 754, using single precision i.e. 32 bits) calculations it's not the case, as sqrt(2) doesn't have an exact representation in binary. So does using a approximated value of sqrt(2) provide different results for left and right hand sides? I...
int i = 3.1 / 2 does not cause any warnings,even with -Wall option.Sometimes,I would like to know where precision lose.Why gcc does not support this warning,while msvc support this one? thanks. EDIT: my gcc -v shows Configured with: ../../gcc-4.4.1/configure --prefix=/mingw --build=mingw32 --enable-languages=c,ada,c++,fortran,objc,o...
Does the SQL Server decimal data type stored the actual number of digits after the decimal for each value in a column? For example UPDATE [Mixes] SET [Concentration] = 0.0012500 WHERE [Id] = 1 where Concentration Decimal(21,11) Is it possible to retrieve the exact "0.0012500" value as entered by the user, or is it necessary to st...