Processing large (over 1 Gig) files in PHP using stream_filter_*

Question

$fp_src=fopen('file','r');

$filter = stream_filter_prepend($fp_src, 'convert.iconv.ISO-8859-1/UTF-8');

while(fread($fp_src,4096)){
    ++$count;
    if($count%1000==0) print ftell($fp_src)."\n";
}

When I run this the script ends up consuming ~ 200 MB of RAM after going through just 35MB of the file.

Running it without the stream_filter zips right through with a constant memory footprint of ~10 MB.

What gives?

Answer 1

From what I'm reading here, you are not implementing stream_filter_prepend() correctly, although there could be something I misunderstand about the process.

Als, I'm not totally sure, but I'm willing to bet that this has more to do with the fact that iconv is an expensive process, and less to do with the fact that you're using it as stream filter.

Good luck.

Answer 2

You only need to register custom filters. iconv is built in. It's not the particular operation, using a stream filter for rot13 exhibits similar behavior.

Answer 3

Any particular reason you want to use stream_filter_prepend()? If it's causing memory problems, then I'd find another way to do what it does.