phpcrawl

PHPCrawl sometimes returns empty handed

I'm using the PHPCrawl class to spider websites and build a list of links. It all works well, if slowly, and I then use the links to perform other tasks. I'm encountering a problem where the first time I run the script it completes with no result, then the next time I run it it works as expected. It's failing about 30% of the time. I t...