How can I work around a round-off error that causes an infinite loop in Perl's Statistics::Descriptive?

Question

I'm using the Statistics::Descriptive library in Perl to calculate frequency distributions and coming up against a floating point rounding error problem.

I pass in two values, 0.205 and 0.205, (taken from other numbers and sprintf'd to those) to the stats module and ask it to calculate the frequency distribution but it's getting stuck in an infinite loop.

Stepping through with a debugger I can see that it's doing:

my $interval = $self->{sample_range}/$partitions;

my $iter = $self->{min};

while (($iter += $interval) <  $self->{max}) {

  $bins{$iter} = 0;

  push @k, $iter;  ##Keep the "keys" unstringified

}

$self->sample_range (The range is max-min)is returning 2.77555756156289e-17 rather than 0 as I'd expect. This means that the loop ((min+=range) < max)) enters a (for all intents and purposes) infinite loop.

DB<8> print $self->{max};
0.205
DB<9> print $self->{min};
0.205
DB<10> print $self->{max}-$self->{min};
2.77555756156289e-17

So this looks like a rounding problem. I can't think how to fix this on my side though, and I'm not sure editing the library is a good idea. I'm looking for suggestions of a workaround or alternative.

Cheers, Neil

Answer 1

Not exactly a rounding problem; you can see the more precise values with something like

printf("%.18g %.18g", $self->{max}, $self->{min});

Looks to me like there's a flaw in the module where it assumes the sample range can be divided up into $partitions pieces; because floating point doesn't have infinite precision, this isn't always possible. In your case, the min and max values are exactly adjacent representable values, so there can't be more than one partition. I don't know what exactly the module is using the partitions for, so I'm not sure what the impact of this may be. Another possible problem in the module is that it is using numbers as hash keys, which implicitly stringifies them which slightly rounds the value.

You may have some success in laundering your data through stringization before feeding it to the module:

$data = 0+"$data";

This will at least ensure that two numbers that (with the default printing precision) appear equal are actually equal.

Answer 2

That shouldn't cause an infinite loop. What would cause that loop to be infinite would be if $self->{sample_range}/$partitions is 0.

Answer 3

Hi!

I am the Statistics::Descriptive maintainer. Due to its numeric nature, many rounding problems have been reported. I believe this particular one was fixed in a later version to the one you were using that I released recently, by using multiplication for the divisions instead of +=.

Please use the most up-to-date version from the CPAN, and it should be better.