Large Memory Usage on Small Server (Optimization Question)

Question

I have an application that analyzes data from input files that are generated by our core system. Depending on the client, that file can vary in size (files contain online marketing metrics such as clicks, impressions, etc..). One of our clients has a website that gets a fairly large amount of traffic, and the metric files generated are around 3-4 megabytes in size. This application currently analyzes three files at a time, each file being a different time aggregate.

I'm reading in the file using a CSV iterator, and it stores the contents of the entire file into a multi-dimensional array. The array for one of the particular files is around 16000 elements long, with each subarray being 31 elements. The dataprocessor object that handles loading this data utilizes about 50MB of memory. Currently the PHP memory limit is set to 100MB. Unfortunately the server this application is on is old and can't handle much of a memory increase.

So this brings me to the question: how can I optimize processing a file this size?

Could a possible optimization be reading in parts of the file, calculate, store, repeat?

Answer 1

You're on the right track. If at all possible, read a line, do whatever you need to do to it (counting whatever you're counting, etc), and then discard the line.

See the example for fgets()

Answer 2

Why not simply read the file line by line... -> read line -> store what you need, update your statistics -> read next line, etc.

Answer 3

You could modify CSVIterator and read parts of the file at a time or a line at a time.

$handle = fopen("/tmp/inputfile.txt", "r");
if ($handle) {
  while(!feof($handle)){
    $buffer = fread($handle, 4096);
    echo $buffer;
  }
}

or

$handle = fopen("/tmp/inputfile.txt", "r");
if ($handle) {
  while (!feof($handle)) {
    $buffer = fgets($handle, 4096);
    echo $buffer;
  }
  fclose($handle);
}