How can I prevent memory leak in csv file import / isbn lookup script

Question

My website allows users to upload a csv file with a list of books. The script then reads this file and checks the isbn number against Amazon, using the PEAR Services_Amazon class, returning enhanced book data. However, whenever I run the script on a list of books the amount of memory consumed steadily increases until I get a fatal error. At the moment, with 32 MB allocated, I can only read 370 records of the CSV file before it crashes.

I have a user with a 4500 record file to import and a virtual server with 256 MB of RAM, so increasing the memory limit is not a solution.

Here is a simplified version of the CSV import:

$handle = fopen($filename, "r");
 while (($data = fgetcsv($handle, 1000, ",")) !== FALSE) {
 $isbn = $data[6];
 checkIsbn($isbn);
 }

Here is a trimmed version of the function:

function checkIsbn($isbn) {     
 $amazon = &new Services_Amazon(ACCESS_KEY_ID, SECRET_KEY, ASSOC_ID);
 // -- $options array filled with $isbn, other requested info --
 $products = $amazon->ItemSearch('Books', $options);        
 // -- Then I create an array from the first result --
  $product = $products['Item'][0];
  $title = $product['ItemAttributes']['Title']; 
  // -- etc... various attributes are pulled from the $product array --
 mysql_query($sql); // -- put attributes into our DB
  unset($product); 
  unset($products);
  usleep(1800000); // maximum of 2000 calls to Amazon per hour as per their API
return $book_id;    
 }

What I've tried: unsetting the arrays as well as setting them to NULL, both in the function and in the CSV import code. I have increased all my timeouts to ensure that's not an issue. I installed xdebug and ran some tests, but all I found was that the script just kept increasing in memory each time the Amazon class is accessed (I'm no xdebug expert). I'm thinking that maybe the variables in the Services_Amazon class are not being cleared each time it's run, but have no idea where to go from here. I'd hoped unsetting the two arrays would do it, but no luck.

Edit: Update: I've decided that this may be a problem in the PEAR class (and looking at some of the questions here relating to PEAR, this does seem possible). Anyway, my OOP skills are very few at the moment, so I found a way to do this by reloading the page multiple times - see my answer below for details.

Answer 1

How about only creating a single instance of the $amazon object and passing it in to your checkIsbn function? They you wouldn't need to create 4500 instances.

$amazon = &new Services_Amazon(ACCESS_KEY_ID, SECRET_KEY, ASSOC_ID);
$handle = fopen($filename, "r");
while (($data = fgetcsv($handle, 1000, ",")) !== FALSE) {
 $isbn = $data[6];
 checkIsbn($amazon, $isbn);
}
unset($amazon);

Answer 2

first of all, this is not a memory leak but bad programming... second point is that unset won't free the used memory, it just removes the reference to the variable from the current scope.

also better try to not copy the memory here but just make $produkt and $title a pointer by assigning only the references to $products;

$product = &$products['Item'][0];
$title = &$product['ItemAttributes']['Title']; 

then, instead of only unset() do

$products = NULL;
unset($products);

this will free the memory, not immediately but when the php garbage collector runs the next time...

also why do you create a new instance iof the Serverces_Amazon each time the function i called? what about a class member to create instance in when constructing your object.

class myService
{
    protected $_service;

    public function __construct()
    {
        $this->_service = new Services_Amazon(ACCESS_KEY_ID, SECRET_KEY, ASSOC_ID);
    }

    public function checkIsbn($isbn)
    {
        //...
        $this->_service->ItemSearch('Books', $options);
        //...
    }
}

$myService = new myService;
$handle = fopen($filename, "r");
while (($data = fgetcsv($handle, 1000, ",")) !== FALSE) {

    $bookId = $myService->checkIsbn($data[6]);
}

and furthermore you assume that your users all use the same CSV format which is very unlikely... so better use a real CSV parser which can handle all possible CSV notations...

Answer 3

I think you should also look into how you are connecting to the database - are you creating fresh connections each time checkIsbn is called? That could also be part of the problem.