Is LWP::UserAgent not thread-safe?

Question

I'm running 40-or-so threads with the following subroutine:

my $app = shift;
my $ua = LWP::UserAgent->new();
$ua->timeout(5);
my $response = $ua->get($$app{'watch_url'});
my $new_md5;
if ($response->is_success()) {
    $new_md5 = md5_hex($response->content());
}
return ($$app{'short_name'}, $$app{'watch_md5'}, $new_md5);

Core dumps ensue about 3/4 of the time. LWP and LWP::UserAgent are pure Perl, so I'm caught off-guard by this. Is LWP::UserAgent not thread-safe?

Update:

Here's a minimal version to reproduce the issue:

use strict;
use warnings;
use threads;
use LWP::UserAgent;

sub check_app {
    my $ua = LWP::UserAgent->new();
    $ua->timeout(5);
    $ua->get('http://www.flatdoc.com/?' . rand(10));
}

my @threads;
for (my $i = 0; $i < 40; $i++) {
    my $thread = threads->create(\&check_app);
    push(@threads, $thread);
}
foreach (@threads) {
    $_->join();
}

Answer 1

Non-thread safe pure-Perl code does not cause a segfault (in fact, no pure Perl code should cause a segfault). A bug in Perl causes a segfault. And threads in Perl are historically very buggy, but they've gotten a lot better.

Your code runs fine in 5.10.1, and HTTP::Lite probably just doesn't tickle whatever perl bug you've run into. Likely you just need to use a newer version of Perl. The older and closer to Redhat you get, the less stable threads are. If you're going to use threads, use the newest Perl you can get your hands on.

As an alternative to threads, you can use something like Parallel::ForkManager, LWP::Parallel or even the amazing forks module which emulates threads using fork.