Compute numerical values from a space separated text file, within a range of lines

Question

I have a file with the following values:

for 3 threads:
Average time taken for API1 is: 19097.7 nanoseconds.
Average time taken for API2 is: 19173.1 nanoseconds.
Average time taken for API2 is: 19777.7 nanoseconds.
Average time taken for API2 is: 19243.1 nanoseconds.
Average time taken for API1 is: 19737.7 nanoseconds.
Average time taken for API2 is: 19128.1 nanoseconds.
for 5 threads:
Average time taken for API1 is: 19097.7 nanoseconds.
Average time taken for API2 is: 19173.1 nanoseconds.
Average time taken for API2 is: 19777.7 nanoseconds.
...

I wish to calculate the sum of the 1API lines and the 2API lines, and sum them. Another requirement is that i also want to figure out for each thread on a separate basis. Is there a way to do this using perl, sed, awk or just shell scripts?

What I can current get is:

cat result | grep API1 | awk {'print $7'}

Answer 1

You can use a combination of grep en awk. grep to select only the lines with data (where API is in) and awk for the counting.

grep API file | awk '{ arr[$5]+=$7 } END {for (i in arr) {print i,arr[i]}   } ' -

(change file with filename or remove for reading from stdin)

If you want to calculate the different sums you can do this

awk '{ if($1 == "for") id = $2; else arr[id $5]+=$7 } END {for (i in arr) {print i,arr[i]}   } ' testfile

Output:

5API1 19097.7
5API2 38950.8
3API1 38835.4
3API2 77322

Answer 2

short and unreadable:

perl -lane 'END{&h}sub h{print"\t$_ => $h{$_}"for keys%h;%h=()}&h,print,next if/^for/;$h{$F[4]}+=$F[6]' data

Readable, but must be a script:

#!/usr/bn/perl

use strict;
use warnings;

my %counts;
my $thread = "undefined";
while (<>) {
    if (/^for ([0-9]+)/) {
     $thread = $1;
     next;
    }
    my ($item, $time) = /for (\S+) is: (\S+) nano/;
    $counts{$thread}{$item} += $time;
}

for my $thread (sort { $a <=> $b } keys %counts) {
    print "for $thread threads:\n";
    for my $item (sort keys %{$counts{$thread}}) {
     print "\t$item => $counts{$thread}{$item}\n";
    }
}

Answer 3

I don't understand your last requirement (there are no threads specified), but I'll get you a setup for that information and fulfill the requirement I can understand. The data is broken up so you can get to it. And although I don't understand how you'd use the 'for x threads:' line, it is at least captured, so you might be able to work with it.

use List::Util qw<sum>;

my $fh = FileHandle->new( PATH_TO_DATAFILE );
my $data 
    = { trial_times => []
      , totals      => {}
      };
my $precision = 0;

while ( <$fh> ) { 
    if ( m/^for (\d+) threads:/ ) { 
        push @{$data->{trial_times}}, {};
    }
    elsif ( m/^Average time taken for (API\w+) is: (\d+\.(\d+)) nanoseconds./ ) {
        push @{$data->{trial_times}[-1]{$1}}, $2;
        push @{ $data->{totals}->{$1} }, $2;
        $precision = length $3 if length $3 > $precision;
    }
}

### $data

foreach my $api ( keys %{ $data->{totals} } ) { 
    my @list = @{ $data->{totals}{$api} };
    my $sum  =sum @list;

    printf "Sum for %d runs of API $api: %0.${precision}f (Average: %0.${precision}f)\n"
         , scalar @list, $sum, $sum / scalar @list
         ;
}

my @combined = map { @$_ } values %{$data->{totals}};
### @combined
my $sum      = sum @combined;
printf "Combined %d runs for %0.${precision}f total (Average: %0.${precision}f)\n"
    , scalar @combined, $sum, $sum / scalar @combined
    ;