nginx-tornado-django request timeout

Question

We are using nginx-tornado-django to provide web services. That is, no web page frontend. The nginx server serves as a load-balancer. The server has 8 cores, so we launched 8 tornado-django processes on every server. Memcached is also deployed to gain better performance. The requests per day is about 1 million per server. We use MySQL as backend DB.

The code is tested and correct. Our profiling shows that normally every request are processed within 100ms. The problem is, we find that about 10 percent of the requests suffers from time-out issue. Many requests didn't even reach tornado. I really don't have much experience on tuning of nginx/tornado/MySQL. Right now I don't have a clue on what is going wrong. Any advise is appreiciated.

Answer 1

You may want to separately profile each component and tier: nginx, tornado, django and mysql

I'm not sure about the details of what your application does and it's just a guess, but it's possible the slowdown is happening in the read/write of data to mysql. But your profiling should find determine exactly where the issue is happening.