PHP/Cronjob saved search scalability

Question

Our real estate website sends email notifications when properties come on the market matching users saved search criteria.

I have a PHP script that runs on the cronjob to process the searches.

The system now has over 40,000 users. I ran into an issue where I exhausted the PHP memory size. As the system grows I can continue to increase the memory size in php.ini but would like to find a more robust way to do this.

Is there a more scalable way to accomplish this? Should I build something in a more robust language? Perhaps a threaded application in Java or Python?

Answer 1

Use some identifier in the saved search tables to partition the tasks by the amount of computers dedicated to it.

divide and conquer by farming out sections of the search refactoring to many machines, each running some partition of the tasks.

Each script can do

SELECT * FROM saved_search_tbl WHERE ssid IN CALCRANGE(searchid, node_id)

Where INRANGE is some partitioning logic based on amount of work and the machine's node number. When new searches come in, you cound load balance them across machines by assigning primary and secondary compute nodes.

A few ideas, hope they help!

good luck