In Perl, how can I wait for threads to end in parallel?

Question

I have a Perl script that launches 2 threads,one for each processor. I need it to wait for a thread to end, if one thread ends a new one is spawned. It seems that the join method blocks the rest of the program, therefore the second thread can't end until everything the first thread does is done which sort of defeats its purpose.

I tried the is_joinable method but that doesn't seem to do it either.

Here is some of my code :

use threads;
use threads::shared;

@file_list = @ARGV;      #Our file list
$nofiles = $#file_list + 1; #Real number of files 
$currfile = 1;     #Current number of file to process

my %MSG : shared;              #shared hash

$thr0 = threads->new(\&process, shift(@file_list));
$currfile++;
$thr1 = threads->new(\&process, shift(@file_list));
$currfile++;

while(1){
 if ($thr0->is_joinable()) {
  $thr0->join;
        #check if there are files left to process
  if($currfile <= $nofiles){ 
   $thr0 = threads->new(\&process, shift(@file_list));
   $currfile++;
  }
 }

 if ($thr1->is_joinable()) {
  $thr1->join;
        #check if there are files left to process
  if($currfile <= $nofiles){
   $thr1 = threads->new(\&process, shift(@file_list));
   $currfile++;
  }
 }
}

sub process{
       print "Opening $currfile of $nofiles\n";
       #do some stuff
       if(some condition){
               lock(%MSG);
               #write stuff to hash
       }
       print "Closing $currfile of $nofiles\n";
}

The output of this is :

Opening 1 of 4
Opening 2 of 4
Closing 1 of 4
Opening 3 of 4
Closing 3 of 4
Opening 4 of 4
Closing 2 of 4
Closing 4 of 4

Answer 1

I think you need to move the code that pulls the next file from the list into the threads themselves.

So every thread would not just process one file, but continue to process until the list is empty.

This way, you also save on the overhead of creating new threads all the time.

Your main thread will then join both of them.

Of course, this requires synchronization on the list (so that they do not pull the same data). Alternately, you could split the list into two (one for each thread), but that might result in an unlucky distribution.

(PS: No Perl God, just a humble monk)

Answer 2

First off, a few comments on the code itself. You need to make sure you have:

use strict;
use warnings;

at the beginning of every script. Second:

@file_list = @ARGV;      #Our file list
$nofiles = $#file_list + 1; #Real number of files 

is unnecessary as an array in scalar context evaluates to the number of elements in the array. That is:

$nofiles = @ARGV;

would correctly give you the number of files in @ARGV regardless of the value of $[.

Finally, the script can be made much simpler by partitioning the list of files before starting the threads:

use strict; use warnings;

use threads;
use threads::shared;

my @threads = (
    threads->new(\&process, @ARGV[0 .. @ARGV/2]),
    threads->new(\&process, @ARGV[@ARGV/2 + 1 .. @ARGV - 1]),
);

$_->join for @threads;

sub process {
    my @files = @_;
    warn "called with @files\n";
    for my $file ( @files ) {
        warn "opening '$file'\n";
        sleep rand 3;
        warn "closing '$file'\n";
    }
}

Output:

C:\Temp> thr 1 2 3 4 5
called with 1 2 3
opening '1'
called with 4 5
opening '4'
closing '4'
opening '5'
closing '1'
opening '2'
closing '5'
closing '2'
opening '3'
closing '3'

Alternatively, you can let the threads move on to the next task as they are done:

use strict; use warnings;

use threads;
use threads::shared;

my $current :shared;
$current = 0;

my @threads = map { threads->new(\&process, $_) } 1 .. 2;
$_->join for @threads;

sub process {
    my ($thr) = @_;
    warn "thread $thr stared\n";

    while ( 1 ) {
        my $file;
        {
            lock $current;
            return unless $current < @ARGV;
            $file = $ARGV[$current];
            ++ $current;
        }
        warn "$thr: opening '$file'\n";
        sleep rand 5;
        warn "$thr: closing '$file'\n";
    }
}

Output:

C:\Temp> thr 1 2 3 4 5
thread 1 stared
1: opening '1'
1: closing '1'
1: opening '2'
thread 2 stared
2: opening '3'
2: closing '3'
2: opening '4'
1: closing '2'
1: opening '5'
1: closing '5'
2: closing '4'