views:

76

answers:

1

I have not been capable of finding this anywhere online. I was looking to find out using a profiler how to better optimize my code, and when sorting by which functions use up the most time cumulatively, things like str(), print, and other similar widely used functions eat up much of the profile. What is the best way to profile a python program to get the user-defined functions only to see what areas of their code they can optimize?

I hope that makes sense, any light you can shed on this subject would be very appreciated.

+2  A: 

OK, I assume your real goal is to make your code as fast as reasonably possible, right?

It is natural to assume you do that by finding out how long your functions take, but there is another way to look at it.

Consider as your program runs it traces out a call tree, which is kind of like a real tree outside your window. The trunk is like the main function, and where any branch splits off from it is like calling another function.

Suppose each "leaf" takes a certain amount of time, and what you want to do is prune the tree so as to remove as many leaves as possible.

One way is to find branches with lots of leaves and cut off the leaves. Another way is to cut off whole branches if you don't need them. The problem is to find heavy branches that you don't need.

One bone-simple way to do this is to pick several leaves at random, like 10, and on each one, trace a line down its branch all the way back to the trunk. Any branch point will have some number of these lines running through it, from leaf to trunk. The more lines run through that branch point, the more leaves are on that branch, and the more you could save by pruning it.

Here's how you can apply this to your program. To sample a leaf, you pause the program at random and look at the call stack. That is the line back to the trunk. Each call site on it (not function, call site) is a branch point. If that call site is on some fraction of samples, like 40%, then that is roughly how much you could save by pruning it.

So, don't think of it as measuring how long functions take. Think of it as asking which call sites are "heavy". That's all there is to it.

Mike Dunlavey
What an interesting approach to investigating program behavior. Perhaps not novel, but it really drives home the question "what is my program ACTUALLY doing?" Thanks!
stw_dev
@stw_dev: You're right, it is *way* not new, just not taught, for whatever reason.
Mike Dunlavey
@Mike: Do you have to use gprof to do so? I am currently on a Windows Machine for the code I want to Optimize. Or does CProfile still work? But great explanation after I fully wrap my brain around it. I forgot to mention I am using Python(not the greatest language when it comes to speed)
Tim McJilton
@Tim McJilton: you wouldn't use gprof. Python has a debugger called `pdb` that is very much like `gdb`. I'm pretty sure you can pause it with ctrl-C or ctrl-Break. Then print the stack trace. Python has profilers, but I don't think any of them actually do this.
Mike Dunlavey
@Mike Dunlavey: Just because I am new to this kind of profiling, I am going to see if I can get this straight. The idea is to randomly stop the program and see what current functions it is in each time, and then after a few times, count the number of times it is in each function, and which every one is greatest try to cut it out//minimize it to speed it up? My main question is whether there would be some easy way to embed this in the code or program? And how do you get a truly random sample? Thanks for your help Mike! Went to your wiki page and it seems that this is is your area of expertise
Tim McJilton
Mike Dunlavey
@Mike Dunlavey: I just realized I don't think your trick will work with the specific problem I am working on. I have a lot of reading/writing to a specific hard drive going on, which is a good amount of the over-head. That being said, most likely I will interrupt that situation more than a poorly coded area. Though the trick seems helpful.
Tim McJilton
@Tim McJilton: It's OK, because it will tell you what percent of time you're spending in that I/O, just like anything else that takes time. It's a good way to detect unnecessary I/O. Then when all I/O is necessary, just discard samples that occur during I/O, and you do what you can to tune the remaining time. If the remaining time doesn't amount to much (10-20%) then you're pretty close to optimal.
Mike Dunlavey
Mike Dunlavey
Mike I think your trick will be Good for other samples. I am writing functions to put as much strain on the drive as possible using low level drive commands. Though one thing I discovered for python programming that i wanted to mention for anyone that may dig this up. RunSnakeRun is a good python based profile reader. Makes is easy to visualize what files are calling what functions for how long. Thanks again Mike.
Tim McJilton
@Tim McJilton: You're welcome. I wish I could be so sanguin about it. There is a firmly entrenched set of myths that have thoroughly infected (almost) all profilers and users of profilers (http://stackoverflow.com/questions/1777556/alternatives-to-gprof/1779343#1779343). It's no accident that certain questions about profiling and performance keep showing up on SO over and over and over, because those myths keep rubbing up against reality.
Mike Dunlavey