These lines in C# decimal a = 2m; decimal b = 2.0m; decimal c = 2.00000000m; decimal d = 2.000000000000000000000000000m; Console.WriteLine(a); Console.WriteLine(b); Console.WriteLine(c); Console.WriteLine(d); Generates this output: 2 2.0 2.00000000 2.000000000000000000000000000 So I can see that creating a decimal variable from a ...
Yesterday I asked this general question about decimals and their internal precisions. Here is a specific question about the scenario I'm trying to address. I have column in SqlServer that is typed - decimal(18,6). When I fetch these values, the .net decimals that are created match the precision in the database. They look like this: 1...
How many distinct numbers are from 1.5 x 10-45 to 3.4 x 1038 (IEE754 single precision floats)? ...
For a simple project I have to make large numbers (e.g. 4294967123) readable, so I'm writing only the first digits with a prefix (4294967123 -> 4.29G, 12345 -> 12.34K etc.) The code (simplified) looks like this: const char* postfixes=" KMGT"; char postfix(unsigned int x) { return postfixes[(int) floor(log10(x))]; } It works, but...
I'm curious: If you do a printf("%f", number); what is the precision of the statement? I.e. How many decimal places will show up? Is this compiler dependent? ...
I am getting a weird problem while parsing a double value in managed C++. It may be that I am doing something wrong. When I do: double value = 0.006; result = Math::Parse( value) The output of result is 0.006000000000001. Why it is appending a 1? Also when I go an round the value to 5 decimal places, it fails. I am doing: result...
Now() in VBScript appears to return time in 10,000,000th of a second precision when called as CDbl(Now()). In attempting to use this to write a more accurate implementation of now which returns CIM_DATETIME format I found that in VBScript, despite being particularly precise, is not very accurate with the time only updating once per secon...
I am doing high precision scientific computations. In looking for the best representation of various effects, I keep coming up with reasons to want to get the next higher (or lower) double precision number available. Essentially, what I want to do is add one to the least significant bit in the internal representation of a double. The ...
I need to compare two datetime values to determine equality(exactly the same),using minute precision.Would this be the best way to do it? My dates could have seconds and milliseconds, but i want to consider only down till minutes. where (Math.Abs(datetime1.Subtract(datetime2).TotalMinutes) == 0) ...
Imagine I write a simple calculator application, that just calculates simple stuff like 1.5 + 30 + 9755 - 30 - 20000 + 999900.54 I remember slightly that there were some precision problems when using floating point numbers. At which point would my calculator app start to create wrong results? Most of the time, I would just calcula...
I can only assume this is a bug. The first assert passes while the second fails: double sum_1 = 4.0 + 6.3; assert(sum_1 == 4.0 + 6.3); double t1 = 4.0, t2 = 6.3; double sum_2 = t1 + t2; assert(sum_2 == t1 + t2); If not a bug, why? ...
I want to store a number with the following 0.000 which is the best data type to use. A double? Also I guess an INT is just out of the question? ...
I was testing this code from Brainteasers: double d1 = 1.000001; double d2 = 0.000001; Console.WriteLine((d1 - d2) == 1.0); And the result is "False". When I change the data type: decimal d1 = 1.000001M; decimal d2 = 0.000001M; decimal d3 = d1-d2; Console.WriteLine(d3 == 1...
I'm interested in implementing an algorithm on the GPU using HLSL, but one of my main concerns is that I would like a variable level of precision. Are there techniques out there to emulate 64bit precision and higher that could be implemented on the GPU. Thanks! ...
An MS SQL Server 2008 BULK INSERT to a datetime column introduces an error in the 3rd decimal place of the seconds portion: 2009-09-19 15:02:41.328 in the input file becomes 2009-09-19 15:02:41.327 in the database. Here is what the input text file contains: 1 2009-09-19 15:02:41.328 Here is what the table looks like after BULK INS...
Hello, Consider the following C code: int c=((0xa3>>6)&0x1f)|0xc0; printf("%d\n",c); It correctly prints out 194 (0xc2). If I write the same thing in wcalc, the result is 195, or 0xc3. Is this some sort of precision error? Is this expected behavior? floor and ceiling functions do not work... or, to be more specific, floor works f...
I would like to approximate the value of e to any desired precision. What is the best way to do this? The most I've been able to get is e = 2.7182818284590455. Any examples on a modification of the following code would be appreciated. public static long fact(int x){ long prod = 1; for(int i = 1; i <= x; i++) prod = prod * i; retu...
I would like to display a percentage with three decimal places unless it's greater than 99%. Then, I'd like to display the number with all the available nines plus 3 non-nine characters. How can I write this in Python? The "%.8f" string formatting works decently, but I need to keep the last three characters after the last string of nine...
I expected the following code to produce: "Both are equal", but I got "Both are NOT equal": float a=1.3f; double b=1.3; if(a==b) { System.out.println("Both are equal"); } else{ System.out.println("Both are NOT equal"); } What is the reason for this? ...
I'm wondering what the precision of the Timer class is in System.Timers, because it's a double (which would seem to indicate that you can have fractions of milliseconds). What is it? ...