how to perform profiling for a website?

Question

I currently have a django site, and it's kind of slow, so I want to understand what's going on. How can I profile it so to differentiate between:

• effect of the network
• effect of the hosting I'm using
• effect of the javascript
• effect of the server side execution (python code) and sql access.
• any other effect I am not considering due to the massive headache I happen to have tonight.

Of course, for some of them I can use firebug, but some effects are correlated (e.g. javascript could appear slow because it's doing slow network access)

Thanks

Answer 1

Take a look at the Django debug toolbar - that'll help you with the server side code (e.g. what database queries ran and how long they took); and is generally a great resource for Django development.

The other non-Django specific bits you could profile with yslow.

Answer 2

client side:

• check with firebug if/which page components take long to load, and how long the browser needs to render the page after loading is completed. If everything is fast but rendering takes its time, then probably your html/css/js is the problem, otherwise it's server side.

server side (i assume you sit on some unix-alike server):

• check the web server with a small static content (a small gif or a little html page), using apache bench (ab, part of the apache webserver package) or httperf, the server should be able to answerat least 100 requests per second (of course this depends heavily on the size of your test content, webserver type, hardware and other stuff, so dont take that 100 to seriously). if that looks good,

• test django with ab or httperf on a "static view" (one that doesnt use a database object), if thats slow it's a hint that you need more cpu power. check cpu utilization on the server with top. if thats ok, the problem might be in the way the web server executes the python code

• if serving semi-static content is ok, your problem might be the database or IO-bound. Database problems are a wide field, here is some general advice:

  • check i/o throughput with iostat. if you see lot's of writes then you have get a better disc subsystem, faster raid, SSD hard drives .. or optimize your application to write less.
  • if its lots of reads, the host might not have enough ram dedicated as file system buffer, or your database queries might not be optimized
  • if i/o looks ok, then the database might be not be suited for your workload or not correctly configured. logging slow queries and monitoring database activity, locks etc might give you some idea

if you let us know what hardware/software you use i might be able to give more detailed advice

edit/PS: forgot one thing: of course your app might have a bad design and does lots of unnecessary/inefficient things ...

Answer 3

There are various tools, but problems like this are not hard to find because they are big.

You have a problem, and when you remove it, you will experience a speedup. Suppose that speedup is some factor, like 2x. That means the program is spending 50% of its time waiting for the slow part. What I do is just stop it a few times and see what it's waiting for. In this case, I would see the problem 50% of the times I stop it.

First I would do this on the client side. If I see that the 50% is spent waiting for the server, then I would try stopping it on the server side. Then if I see it is waiting for SQL queries, I could look at those.

What I'm almost certain to find out is that more work is being requested than is actually needed. It is not usually something esoteric like a "hotspot" or an "algorithm". It is usually something dumb, like doing multiple queries when one would have been sufficient, so as to avoid having to write the code to save the result from the first query.

Here's an example.

Answer 4

First things first; make sure you know which pages are slow. You might be surprised. I recommend django_dumpslow.