PHP Memory Management

Question

I have a few time-consuming and (potentially) memory-intensive functions in my LAMP web application. Most of these functions will be executed every minute via cron (in some cases, the cron job will execute multiple instances of these functions).

Since memory is finite, I don't want to run into issues where I am trying to execute a function the environment can no longer handle. What is a good approach at dealing with potential memory problems?

I'm guessing that I need to determine how much memory is available to me, how much memory each function requires before executing it, determine what other functions are being executed by the cron AND their memory usage, etc.

Also, I don't want to run into the issue where a certain function somehow gets execution priority over other functions. If any priority is given, I'd like to have control over that somehow.

Answer 1

You can find out how much memory is currently in use by your script using memory_get_usage But you can not determine how much your next function will need, before executing it. You can only see after execuiting, using memory_get_usage. You can however store the memory your fucntion used the last times in a database and calculate with the average memory amount.

Regarding the eecution priority, I don't think it is posible to determine with PHP. Apache (or whatever webserver you are using) spawns multiple processes and the operating system schedules which one will be executed in which order.

Answer 2

Part of your problem may be the fact you are doing a cron every minute? Why not set some flags so only one instance of that cron is running before another executes the full logic? i.e. create a flat file thats deleted at the end of the cron to act as a 'lock'. This will make sure one cron process fully completes before any others go forward. However, I urge you to refer to my comment on your post so that I and others can give you more solid advice.

Answer 3

Try optimizing your algorithms. Like...

• Once you're finished with a variable you should destroy it if you no longer need it.
• Close MySQL connections after you've finished with them.
• Use recursion.

Also as Jauzsika said change your memory limit in your php.ini, although don't make it too high. If you need more than 256MB RAM then I would suggest changing to a different language instead of PHP.

Answer 4

you could look into caching technologies like APC which lets you write stuff right into the RAM so that you can access it fast which of use if you dont want to do expensive tasks like mysql queries repeatedly.

an example for caching i could think of would be that you could cache emails rather than retreiving them again and again from the email server. basicaly ram caching is a very useful technique if you have things in your script that you want to preserve for the next time of script execution but if your script does unique things every time it is executed it would be useless. also as for contoll you could call memory_get_usage() on each script execution and write that value into the apc cache so that every cron could retreive that value and look whether enough memory is free for it to complete.

as for average usage you could write an array with the last lets say 100 function executions and when you call that function again it could apc_fetch that from the ram and calculate the average memory usage for that function then compare it to how much ram is being used right now and then decide wheter to start. furthermore it could write that estimate into the current memory usage variable to prevent other scripts from being run. at the end of that function you subtract that amount from the variable again. tl;dr: look into the apc_fetch, apc_store and memory_get_usage functions

Answer 5

In your position, I'd consider writing a daemon instead of relying on cron. The daemon could monitor a queue and be aware of the number of child processes it has running. Managing multiple processes definitely isn't php's biggest strength, but you can do it. Pear even includes a System_Daemon package.

Your daemon could use memory_get_usage and call out out free, uptime, and friends to throttle the number of workers to match system conditions.

I don't have any direct experience with this, and I wouldn't be too too surprised if a daemon written in PHP gradually leaked memory. But if it's acceptably slow, cron could cycle the daemon every so often...