PHP array include performance

Question

What type of performance hit can I expect if I include a huge PHP array? For example, lets say I have a 1GB PHP array in "data.php" that looks like

$data = array( //1GB worth of data )

If I include that huge "data.php" file on "header.php", how will it affect the performance of "header.php" when it executes? Thanks!

Answer 1

Depends on your server, CPU, ram, webserver, hard drives, etc.

But most probably: it is not a good idea. it would be much better to read the file using fread or something.

Answer 2

Firstly, you'll need to increase the max memory for your pages to over a gigabyte because that array will be loaded into memory on every page. That means 1GB of data will need to be read and parsed on every script. Best case? That adds 10+ seconds to every page load. More likely, the page won't load at all.

Store it in a database and query it as you need it. I can't imagine a scenario where you'll need that entire array (or a significant part thereof) of that array on every (or, in fact, any) page.

Answer 3

Humm... A lot?

data.php will be huge, so it will take quite a while to read the file due to disk I/O. Then it has to stay on memory so even if you don't have memory restrictions it'll affect performance.

Another bottleneck to consider is the max_execution_time limit. You're most probably doing something wrong if you need 1GB of data in memory... Have you considered storing the raw (padded) data, one element per each line and then just request some specific bytes of that file instead?

---

Example (Write):

$values = array
(
    0 => '127.0.0.1', // 9 chars
    1 => '127.0.0.2', // 9 chars
    2 => '...', // 3 chars
    3 => '255.255.255.255', // 15 chars - this is the max in our set
);

foreach ($values as $key => $value)
{
    // lets pad each value to 15 bytes
    $values[$key] = str_pad($value, 15, ' ', STR_PAD_LEFT);
}

file_put_contents('./test.data', implode('', $values), LOCK_EX);

---

Example (Read):

echo ltrim(file_get_contents('./test.data', false, null, 0 * 15, 15)); // 127.0.0.1
echo '<hr />';
echo ltrim(file_get_contents('./test.data', false, null, 1 * 15, 15)); // 127.0.0.2
echo '<hr />';
echo ltrim(file_get_contents('./test.data', false, null, 2 * 15, 15)); // ...
echo '<hr />';
echo ltrim(file_get_contents('./test.data', false, null, 3 * 15, 15)); // 255.255.255.255

Answer 4

If you store 1G of data on data.php, then it takes a long time to parse the 1GB file then put the array's data to memory, takes time and space. A server handles multiple requests at one time, that is 1GB times 100 to 1000 of total memory depending on your site's daily view.

Why not try to split it to multiple files or use a database? Or even, create a background service that has the hard-coded 1GB data in C, and PHP calls the service to extract parts of the data.

Answer 5

Just a shot in the dark, but seeing as you can't use memcached etc, Have you considered using a disk based cache, either write your own class. So where you would use memcached->storeValue (or whatever the call is), you would instead call your own function which would write the data to a file. The intricacies of this probably lie outside the scope of this question. Another alternative would be to use something already written such as Zend Framework and its various cache classes which should allow you to disk based caching. Other frameworks probably have thier own caching systems which can write to disk (codeIgniter, CakePHP etc). Third alternative would be cache generated data into a mysql databasem while this might seem a little mad, if done properly it would still give you a speed boost in roughly the same way any other caching would. When caching its worth spending a fair bit of time benchmarking to make sure that you are actually removing a bottleneck by doing do it.