Why doesn't the .* consume the entire string in this Perl regex?

Question

Why doesn't the first print statement output what I expect:

first = This is a test string, sec = This is a test string

Since both * and + are greedy, why does the the inner * i.e. inside the "((" in the first match not consuming the entire string?

use strict;
use warnings;

my $string = "This is a test string";
$string =~ /((.*)*)/; 
print "first = $1, sec = $2\n";  #prints "first = This is a test string, sec ="

$string =~ /((.+)*)/;
print "first = $1, sec = $2\n";  #prints "first = This is a test string, sec = This is a test string"

Answer 1

In the first regex .* is matched two times. The first time it matches the whole string. The second time it matches the empty string at the end, because .* matches the empty string when there is nothing else to match.

This does not happen with the other regex because .+ can't match the empty string.

Edit: As to what goes where: $2 will contain what is matched the last time .* / .+ are applied. $1 will contain what is matched by (.*)* / (.+)*, i.e. the whole string.

Answer 2

The problem with the first regex is a combination of the fact that ()* only saves the last match and .* matches an empty string (i.e. nothing). So, given

"aaab" =~ /(.)*/;

$1 will be "b". If you combine that behavior with the fact that .* matches an empty string, you can see that there are two matches of the inner capture: "This is a test string" and "". Since the empty string came last it gets saved to $2. $1 is the whole capture, so it is equivalent to "This is a test string" . "". The second case works as you expect it to because .+ will not match an empty string.

Answer 3

I don't have an answer, but I do have different way of framing the issue, using simpler and perhaps more realistic regular expressions.

The first two examples behave exactly as I expect: .* consumes the entire string and the regular expression returns a list with only one element. But the third regular expression returns a list with 2 elements.

use strict;
use warnings;
use Data::Dumper;

$_ = "foo";
print Dumper( [ /^(.*)/g ] ); # ('foo')     As expected.
print Dumper( [ /.(.*)/g ] ); # ('oo')      As expected.
print Dumper( [ /(.*)/g  ] ); # ('foo', '') Why?

Many of the answers so far have emphasized that .* will match anything. While true, this response does not go to the heart of the matter, which is this: Why is the regular expression engine still hunting after .* has consumed the entire string? Under other circumstances (such as the first two examples), .* does not throw in an extra empty string for good measure.

Update after the useful comments from Chas. Owens. The first evaluation of any of the three examples results in .* matching the entire string. If we could intervene and call pos() at that moment, the engine would indeed be at the end of the string (at least as we perceive the string; see the comments from Chas. for more insight on this). However, the /g option tells Perl to try to match the entire regex again. That second attempt will fail for examples #1 and #2, and that failure will cause the engine to stop hunting. However, with regex #3, the engine will get another match: an empty string. Then the /g option tells the engine to try the entire pattern yet again. Now there really is nothing left to match -- neither regular characters nor the trailing empty string -- so the process stops.