Best way to profile/optimize a website on google's appengine

Question

I'm currently trying to optimize my website, which run on the google's appengine. It's not an easy task, because I'm not using any powerful tool.

Does anyone have experience in optimizing python code for this purpose? Have you find a good python profiler?

Answer 1

The python site listed 3 profilers to choose from: http://docs.python.org/library/profile.html

Answer 2

I have found Gprof2Dot extremely useful. The output of the profiling modules I've tried as pretty unintuitive to interpret.

Gprof2Dot turns the cProfile output into a pretty looking graph, with the slowest chain(?) highlighted, and a bit of information on each function (function name, percentage of time spend on this function, and number of calls).

An example graph (1429x1896px)

I've not done much with the App Engine, but when profiling non-webapp scripts, I tend to profile the script that runs all the unittests, which may not be very accurate to real-world situations

One (better?) method would be to have a script that does a fake WSGI request, then profile that.

WSGI is really simple protocol, it's basically a function that takes two arguments, one with request into, one with a callback function (which is used for setting headers, among other things). Perhaps something like the following (which is possible-working pseudo code)...

class IndexHandler(webapp.RequestHandler):
    """Your site"""
    def get(self):
        self.response.out.write("hi")

if __name__ == '__main__':
    application = webapp.WSGIApplication([
        ('.*', IndexHandler),
    ], debug=True)

    # Start fake-request/profiling bit
    urls = [
        "/",
        "/blog/view/hello",
        "/admin/post/edit/hello",
        "/makeanerror404",
        "/makeanerror500"
    ]

    def fake_wsgi_callback(response, headers):
        """Prints heads to stdout"""
        print("\n".join(["%s: %s" % (n, v) for n, v in headers]))
        print("\n")

    for request_url in urls:
        html = application({
        'REQUEST_METHOD': 'GET',
        'PATH_INFO': request_url},
        fake_wsgi_callback
        )
        print html

Actually, the App Engine documentation explains a better way of profiling your application:

From http://code.google.com/appengine/kb/commontasks.html#profiling:

To profile your application's performance, first rename your application's main() function to real_main(). Then, add a new main function to your application, named profile_main() such as the one below:

def profile_main():
    # This is the main function for profiling 
    # We've renamed our original main() above to real_main()
    import cProfile, pstats
    prof = cProfile.Profile()
    prof = prof.runctx("real_main()", globals(), locals())
    print "<pre>"
    stats = pstats.Stats(prof)
    stats.sort_stats("time")  # Or cumulative
    stats.print_stats(80)  # 80 = how many to print
    # The rest is optional.
    # stats.print_callees()
    # stats.print_callers()
    print "</pre>"

[...]

To enable the profiling with your application, set main = profile_main. To run your application as normal, simply set main = real_main.

Answer 3

In case anyone is coming here late, check out FirePython, it will handle generating the dot graphs with gprof2dot serverside and you can view them in FireBug.

Answer 4

For profiling the API calls, Guido van Rossum released a library called Appstats that will record and display a lot of good stuff about your app.

You can get the library here: https://sites.google.com/site/appengineappstats/

I wrote an article about it on my blog (with some screenshots): http://blog.dantup.com/2010/01/profiling-google-app-engine-with-appstats