Email Alerts on saved searches, procedure and safety/performance tips&tricks?

Question

I built an email alert for my users (now are only 2,000) so every night a crontab execute a php script that query the mysql to find matches with user's saved search. it's a classified website in my case, but i would like to learn in case i had to build something for bigger clients

my concerns are:

1. what happen if my user grow x10 or x100 times? is the server going to crash? there any tip you can suggest on manage something like that?

2. there is any way to protect my file cron/nightly_script.php to be executed form outside calling it in the url of the browser? consider tham im using a string in crontab like:

   lynx [absolute url/script.php]

3. what about the email blast? for each query if the query has results the script sends an email, so it means a blast of emails...is it going to be considered spam automatically and then i could blacklisted?

thanks!!!

Answer 1

1. Well, you should probably modify your script so that you can spread the load. For example, you can have the cron run 4+ times a day and each time it does a percentage of the user base, instead of doing them all once a day.

2. You can take it out of the web server target path and put the cron somewhere that i not accessible externally. It could be executed like this: php /location/of/script.php

3. I guess it will vary depending on who you send it to, but you should consider how often you send this notice.

Answer 2

As for number 2, the best solution would be to move the script outside the document root so it's not accessible from a browser and call it directly

php [location/script.php]

If you can't do that, I would do a IP check and only allow it to be called from localhost IP.

You could also build in safe checks: store when the last time you sent a email to a particular user was and check that before sending another. This would protect against crontab problems as well as hackers.

Answer 3

Number 1: Monitor the server, watch the load and the time it takes to run. It shouldn't crash it but you may find you get to the point where the load is to high and requests for web pages start to slow down.

But one thing to watch is PHP's memory garbage can be odd sometimes, so watch memory usage of the cron job. If it gets to high PHP will crash.

If it starts to get to much there are lots of solutions; there is no need to have the web server and the email sending on the same machine for instance. As long as they can access the same DB, set up a 2nd server just for email sending. This is what cloud computing is perfect for, hire a 2nd server 4 hours a night (or whatever) and turn it off the rest of the time.

That's just one suggestion ... there are many solutions and it really depends on your situation.

Answer 4

what happen if my user grow x10 or x100 times? is the server going to crash? there any tip you can suggest on manage something like that?

Your server could crash/get slow as hell because of extensive memory/cpu usage. You should use a message queue like redis/beanstalkd/gearmand to throttle your email alerts. My preference goes out to redis. use the blocking pop/push with predis library which support blocking pop/push.

there is any way to protect my file cron/nightly_script.php to be executed form outside calling it in the url of the browser? consider tham im using a string in crontab like:

Don't use cron if you want to scale. Instead create couple of daemons.

• 1 to schedule sending messages(this part could also be cron) to message queue,
• 1 to process messages send to message queue.

Daemons don't need to be spawned each time and spawning processes is (relative) expensive. Second your script should not call any URL anymore but instead call the PHP scripts directly(CLI).

what about the email blast? for each query if the query has results the script sends an email, so it means a blast of emails...is it going to be considered spam automatically and then i could blacklisted?

When using a message queue you can throttle yourself!