As far as I can tell there are a number of things wrong here. First, I would use preg_replace instead of preg_match. So I would write the function like this:
function test_function($content)
{
$txt = '
<div id="\3_container" class="overlay">
<img width="\1" height="\2" src="\3_thumb.jpg" />
</div>';
return preg_replace('%\[custom w="(\d+)" h="(\d+)"\]((?:[a-z][a-z][0-9]+[a-z0-9]))\[\/custom\]%', $txt, $content);
}
Which is basically 2 lines of code. Some other problems you're running into:
1) Your regex fails. The first [ isn't escaped, and as you know that's a special character in regexes. I suspect there are other problems with your regex too.
2) From the PHP manual for strpos:
This function may return Boolean FALSE, but may also return a non-Boolean value which evaluates to FALSE, such as 0 or "". Please read the section on Booleans for more information. Use the === operator for testing the return value of this function.