[PHP] - Lowering script memory usage in a "big" file creation

Question

Hi there people, it looks like I'm facing a typical memory outage problem when using a PHP script.

The script, originally developed by another person, serves as an XML sitemap creator, and on large websites uses quite a lot of memory.

I thought that the problem was related due to an algorithm holding data in memory until the job was done, but digging into the code I have discovered that the script works in this way:

• open file in output (will contain XML sitemap entries)
• in the loop: ---- for each entry to be added in sitemap, do fwrite
• close file
• end

Although there are no huge arrays or variables being kept in memory, this technique uses a lot of memory.

I thought that maybe PHP was buffering under the hood the fwrites and "flushing" data at the end of the script, so I have modified the code to close and open the file every Nth record, but the memory usage is still the same.... I'm debugging the script on my computer and watching memory usage: while script execution runs, memory allocation grows.

Is there a particular technique to instruct PHP to free unsed memory, to force flushing buffers if any?

Thanks

Answer 1

There has to be some place where data is coming from, take a good look at for each entry to be added in sitemap part of code. It might be that huge DB recordset is loaded at once, or something else, similar.

Anyhow, in order to get around that problem, if you're loading data from a database, try limiting number of results, and then looping to get set after set of data.

Answer 2

To flush the file buffers use fflush() (http://ch.php.net/manual/en/function.fflush.php) and to free space allocated to variables a unset() (http://ch.php.net/unset) should do it.

You said you tried to close and reopen the file during the process, so flushing the buffer surely isn't the solution. Why not show us some code, memory leaks are sometimes quite unobvious?

Answer 3

Change the way the XML file is parsed. I'm guessing it's loading the whole tree into memory.

Use an event parser instead. I've handled XML files with several gigabytes with this.

Answer 4

So you are looking for a memory leak in a large PHP program not written by yourself? After you checked the common problems (loading huge db result, not flushing / closing file) without any luck I think you should use a profiler like XDEBUG ( http://xdebug.org/ ) which helps you identify the memory problems. Anything other is just guessing most of the time. I had this experience already a few times...

Answer 5

Here is the code:

link text

Basically