What are the usual bottlenecks (And what tends to break first) to Lamp based sites on EC2 when your number of users increase?
Assuming: -Decent DB design -There are some Ram and CPU intensive processes on cron but no ram/cpu intensive stuff during normal use.
You really need to tell us more about your application. What breaks depends entirely on how it uses resources.
Since you've switched to lighttpd, the webserver itself is going to use fewer resources than apache would, but Apache is rarely the bottleneck unless you've run out of ram or seriously misconfigured it.
Have you tried actually testing your application using ab? Load it up and see what happens.
Good question - we replaced the A with Nginx, our PHP is fpm'd now. And that allows us to setup more app balancers to handle traffic spikes and all that. We also moved the main database to CouchDB (BigCouch) but generally there is no recipe to avoid disaster without knowing what your application does.
EC2 bottlenecks or issues are easier to generalize and pin down.
E.g., a very general bottleneck is the disk i/o.
Even though EBS is faster than the instance storage and also persistence, it's also slow. There are ways to get more EBS performance using RAID setups, but they'll never get you near the speed of SAS.
Another bottleneck is internal network latency. You shouldn't rely on anything being instant, and I guess that's the general rule of thumb with cloud computing. It really is eventually consistent, which also requires your app to adjust to that and behave different.
Last but not least - capacity errors. They happen - e.g., you can't start another instance in the same zone. I've also had instances reboot themselves or disappear. All these things happen in the cloud and need to be dealt with.
The biggest change when moving to EC2 is to let go of actual servers and automate instance bootstrapping. Before I went to the DC for half a day and racked new hardware, installed servers, etc..
Being able to start up and terminate application servers, loadbalancers etc. is the biggest change and also the greatest advantage of the cloud. It helps you to deal with many, many issues easily.