I wanted to display a number with 2 decimal places. I thought I could use toPrecision(2) in JavaScript . However, if the number is 0.05, I get 0.0500. I'd rather it stay the same. What is the best way to do this? I can think of coding a few solutions, but I'd imagine (I hope) something like this is built in? Thanks Update See it o...
Hi, I am trying to use scipy.integrate.quad to integrate a function over a very large range (0..10,000). The function is zero over most of its range but has a spike in a very small range (e.g. 1,602..1,618). When integrating, I would expect the output to be positive, but I guess that somehow quad's guessing algorithm is getting confus...
what's the purpose/point of the REAL() Datatype in mysql? You have float,double and decimal which i understand but where does the "real" datatype come into it? Any ideas? Thank you in advance ;-) is real the same as float or double? ...
I am using SSIS to grab the results from 2 Views in a SQL Server database, union them (with appropriate mapping), filter them and place them into an Access database. This process is working, but the precisions of certain data are changing. I was already aware of the normal float/real problem of storing approximate values (e.g. http://s...
I just read this nice answer given on how to compare floating-point values for equality. The following (slightly modified by me) is suggested instead of straight-forward comparison to 0: const double epsilon = 1e-5; double d = ...; if (Math.Abs(d) < epsilon) { // d is considered equal to 0. } My question is about the name of the ...
We have an interesting problem were we need to determine the decimal precision of a users input (textbox). Essentially we need to know the number of decimal places entered and then return a precision number, this is best illustrated with examples: 4500 entered will yield a result 1 4500.1 entered will yield a result 0.1 4500.00 entered ...
Is it possible to get more decimal precision than the decimal type in C#? According to MSDN, the decimal type can store up to 29 significant digits, however I am wondering if it possible to store more than that and how to go about doing so. Thanks in advance. ...
I have the following code, for an embedded platform where an int is 16 bits and a long int is 32 bits: #define MULTIPLIER 0x1000 static void my_function(uint16_t i, void *p) { uint32_t start = MULTIPLIER * i; ... } My compiler gives me the warning: Warning 1 : lower precision in wider context: '*' for this line. What does...
I am experimenting with a program that inserts data into an SQL 2005 Server database (on XP SP3) at high rate of speed. (This is for collecting timing data so I can evaluate different aspects of my design). My basic set up involves inserting a data into a table like the following (and using an SP that just specifies the payload field...
I'm working on Project Euler to brush up on my C++ coding skills in preparation for the programming challenge(s) we'll be having this next semester (since they don't let us use Python, boo!). I'm on #16, and I'm trying to find a way to keep real precision for 2¹°°° For instance: int main(){ double num = pow(2, 1000); printf("...
Hi, Is there a way to fire events in C# at a resolution of a few microseconds? I am building a midi sequencer and it requires an event to be fired every midi tick which will then play any note registered at that time. At 120 beats per minute and at a resolution of 120 ppqn (pulses per beat/quarter note), that event should fire every 4....
Hi, So I'm just going to dive into this issue... I've got a heavily used web application that, for the first time in 2 years, failed doing an equality check on two doubles using the equality function a colleague said he'd also been using for years. The goal of the function I'm about to paste in here is to compare two double values to...
Hi guys, I was making a program that rounds up numbers with various decimal places, so as an example 2001.3666 would end up as 2001.37, I managed to make this work by adding 0.005 then times 100 and converted to a int and then divided again by 100. Everything worked fine, no issues there, was having some fun making some loops too and ...
Why is it that when I convert a string with value "22.882" to double, using Dbl() it loses precision and is converted to 2288.2? I have to use a double since I'm using the constructor of System.Web.UI.WebControls.Unit (see http://msdn.microsoft.com/en-us/library/ctewx7ch.aspx). ...
Say you have 2 numbers, for each typical mathematical operation is it possible to predict (without significant overhead) whether those operations would result in an overflow of the type which those numbers are currently represented as? ...
I wrote a little program to find the position of a point(x,y) in relation to a line defined by a point(px,py) and an angle(deg) to the x-axis (in a cartesian coordinate system). toRadian deg = deg * (pi / 180) lineSlope deg = tan $ toRadian deg lineYintercept (x,y) deg = y - (x * lineSlope deg) relativePointPosition (px,py) deg (x,y)...
Floating point values are inexact, which is why we should rarely use strict numerical equality in comparisons. For example, in Java this prints false (as seen on ideone.com): System.out.println(.1 + .2 == .3); // false Usually the correct way to compare results of floating point calculations is to see if the absolute difference agains...
Hi all! Imagine that a - b < c (a, b, c are C# doubles). Is it guaranteed that a < b + c? Thanks! EDIT Let's say that the arithmetical overflow doesn't occur unlike the following example: double a = 1L << 53; double b = 1; double c = a; Console.WriteLine(a - b < c); // Prints True Console.WriteLine(a < b + c); // Prints False Imag...
Update OK, after some investigation, and thanks in big part to the helpful answers provided by Jon and Hans, this is what I was able to put together. So far I think it seems to work well. I wouldn't bet my life on its total correctness, of course. public static int GetSignificantDigitCount(this decimal value) { /* So, the decimal t...
$a = '35'; $b = '-34.99'; echo ($a + $b); Results in 0.009999999999998 What is up with that? I wondered why my program kept reporting odd results. Why doesn't PHP return the expected 0.01? ...