PHP & do/while loop memory leaking

Question

I have a do/while loop that goes over database rows. Because it runs many days at the time processing 100000s of rows, memory consumption is important to keep in check or it will crash. Right now every iteration adds about 4kb to script's memory usage. I'm using memory_get_usage() to monitor the usage.

I unset every variable used in the loop first thing in each iteration so I really don't know what else I could do. My guess is that do/while gathers some data with each iteration and this is what consumes the 4kb of memory. I know 4kb doesn't sound like much but it soon starts to add up when you have 100000s of iterations.

Can somebody suggest another way of going through large amount of database rows or how to somehow eliminate this "memory leak"?

edit Here's the UPDATED loop code. Above it is just few require_once()s.

$URLs = new URLs_url(db());
$c = new Curl;
$c->headers = 1;
$c->timeout = 60;
$c->getinfo = true;
$c->follow = 0;
$c->save_cookies = false;

do {
    // Get url that hasn't been checked for a week
    $urls = null;

    // Check week old
    $urls = $URLs->all($where)->limit(10);

    foreach($urls as $url) {
        #echo date("d/m/Y h:i").' | Checking '.$url->url.' | db http_code: '.$url->http_code;

        // Get http code    
        $c->url = $url->url;
        $data = $c->get();

        #echo ' - new http_code: '.$data['http_code'];

        // Save info
        $url->http_code = $data['http_code'];
        $url->lastchecked = time();
        $URLs->save($url);
        $url = null;
        #unset($c);
        $data = null;
        #echo "\n".memory_get_usage().' | ';
        echo "\nInner loop memory usage: ".memory_get_usage();
    }
    echo "\nOuter loop memory usage: ".memory_get_usage();

} while($urls);

Some logs how memory consumption behaves in both loops:

Inner loop memory usage: 611080
Inner loop memory usage: 612452
Inner loop memory usage: 613788
Inner loop memory usage: 615124
Inner loop memory usage: 616460
Inner loop memory usage: 617796
Inner loop memory usage: 619132
Inner loop memory usage: 620500
Inner loop memory usage: 621836
Inner loop memory usage: 623172
Outer loop memory usage: 545240
Inner loop memory usage: 630680
Inner loop memory usage: 632016
Inner loop memory usage: 633352
Inner loop memory usage: 634688
Inner loop memory usage: 636088
Inner loop memory usage: 637424
Inner loop memory usage: 638760
Inner loop memory usage: 640096
Inner loop memory usage: 641432
Inner loop memory usage: 642768
Outer loop memory usage: 556392
Inner loop memory usage: 640416
Inner loop memory usage: 641752
Inner loop memory usage: 643088
Inner loop memory usage: 644424
Inner loop memory usage: 645760
Inner loop memory usage: 647096
Inner loop memory usage: 648432
Inner loop memory usage: 649768
Inner loop memory usage: 651104
Inner loop memory usage: 652568
Outer loop memory usage: 567608
Inner loop memory usage: 645924
Inner loop memory usage: 647260
Inner loop memory usage: 648596
Inner loop memory usage: 649932
Inner loop memory usage: 651268
Inner loop memory usage: 652604
Inner loop memory usage: 653940
Inner loop memory usage: 655276
Inner loop memory usage: 656624
Inner loop memory usage: 657960
Outer loop memory usage: 578732

Answer 1

I think your core problem is that you're only clearing things in the outer loop.

$c = new Curl for instance is going to allocate memory to the heap for each iteration of the inner loop, but you're only unseting the last instance. I'd unset any stuff you can ($c, $data) at the end of the inner loop.

Answer 2

This bit should probably happen only once, before the loop:

$c = new Curl;
$c->headers = 1;
$c->timeout = 60;
...
$c->getinfo = true;
$c->follow = 0;
$c->save_cookies = false;

Edit: Oh, the entire thing is wrapped in a do/while loop. /facepalm

Edit 2: There's also this important bit:

unset($class_object) does not release resources allocated by the object. If used in loops, which create and destroy objects, that might easily lead to a resource problem. Explicitly call the destructor to circumvent the problem.

http://www.php.net/manual/en/function.unset.php#98692

Edit 3:

What is this? Can't this be moved outside of the loop somehow?

$URLs = new URLs_url(db());

Edit 4:

Try removing these lines, for now.

    $url->http_code = $data['http_code'];
    $url->lastchecked = time();
    $URLs->save($url);

Answer 3

The problem is probably

$c = new Curl

Is it possible to instantiate Curl once outside the loop, and then inside keep reusing the same instance. You could reset all fields to null in the loop if you wanted.

I had a similar problem. Unset didn't work - it turned out the garbage collection was rubbish. When I reused objects, it was fine (well, it broke for different reasons so I ended up reimplementing in Java).

Answer 4

This may or may not help you, but way back when in 2000, I had a client who had really slow internet and wanted to do all his website cms updates locally and update to live when done. Back then on IIS on win xp, I could not find a way of increasing script timeout from 60 seconds, and would generally need a good 2 minutes to do the update, so it would obviously time out.

To solve this, I would have the script update a set number of rows which were guaranteed to safely execute in under a minute, then call itself with a parameter of where to continue from, and so on until all rows were updated. Maybe you could try something similar for your situation?

Maybe run it for a set amount of time before calling itself, or in your case, maybe check memory, and redirect when usage gets too high?

I used something like this:

Top of script:

$started = microtime(true);

Then this in your loop:

if((microtime(true)-$started) > ($seconds_to_redirect)) {
    //call script with parameter
}

This is all I can think of.