views:

4137

answers:

4

What other programs do the same thing as gprof?

+7  A: 

Try OProfile. it is much better tool for profiling your code. I would also suggest Intel Vtune.

The two above tools can narrow down time spent in a particular line of code, annotate your code, show assembly and how much particular instruction takes. Beside Time metric, you can also query specific counters, i.e. cache hits, etc.

unlike gprof, you can profile any process/binary running on your system using either of two.

aaa
As also mentioned in the valgrind answer, Zoom from RotateRight ( http://www.rotateright.com ) provides a much nicer interface and allows remote profiling.
JanePhanie
didn't like oprofile, it seemed haphazard
Matt Joiner
@Matt any particular point?
aaa
It wasn't able to cope with more than 10s of execution before generating stat overflows, the output wasn't particularly useful, and the documentation is dreadful.
Matt Joiner
+30  A: 

gprof (read the paper) exists for historical reasons.
If you think it will help you find performance problems, it was never advertised as such. It is only a measurement tool, and only of cpu-bound operations.
Try this instead.
Here's an example.

There's a simple observation about programs. In a given execution, every instruction is responsible for some fraction of the overall time, in the sense that if it were not there, the time would not be spent. During that time, the instruction is on the stack **. When that is understood, you can see that -

gprof embodies certain myths about performance, such as:

  1. that program counter sampling is useful.
    It is only useful if you have an unnecessary hotspot bottleneck such as a bubble sort of a big array of scalar values. As soon as you, for example, change it into a sort using string-compare, it is still a bottleneck, but program counter sampling will not see it because now the hotspot is in string-compare. On the other hand if it were to sample the extended program counter (the call stack), the point at which the string-compare is called, the sort loop, is clearly displayed.

  2. that timing functions is more important than capturing time-consuming lines of code.
    The reason for that myth is that gprof was not able to capture stack samples, so instead it used "instrumentation" which times functions, counts their invocations, and tries to capture the call graph. However, once a costly function is identified, you still need to look inside it for the lines that are responsible for the time. If there were stack samples you would not need to look, those lines would be on the samples.

  3. that the call graph is important.
    What you need to know about a program is not where it spends its time, but why. When it is spending time in a function, every line of code on the stack gives one link in the chain of reasoning of why it is there. If you can only see part of the stack, you can only see part of the reason why, so you can't tell for sure if that time is actually necessary. What does the call graph tell you? Each arc tells you that some function A was in the process of calling some function B for some fraction of the time. Even if A has only one such line of code calling B, that line only gives a small part of the reason why. If you are lucky enough, maybe that line has a poor reason. Usually, you need to see multiple simultaneous lines to find a poor reason if it is there. If A calls B in more than one place, then it tells you even less.

  4. that recursion is a tricky confusing issue.
    That is only because gprof and other profilers perceive a need to generate a call-graph and then attribute times to the nodes. If one has samples of the stack, the time-cost of each line of code that appears on samples is a very simple number - the fraction of samples it is on. If there is recursion, then a given line can appear more than once on a sample. No matter, the time slice represented by the sample is being spent because the line is on it, regardless of how many times it is on it. If the line could be made to take no time (such as by deleting it or branching around it), then the fraction of time it had been on the stack would no longer be spent.

  5. that accuracy of time measurement (and therefore a large number of samples) is important.
    Think about it for a second. If a line of code is on 3 samples out of five, then if you could shoot it out like a light bulb, that is roughly 60% less time that would be used. Now, you know that if you had taken a different 5 samples, you might have only seen it 2 times, or as many as 4. So that 60% measurement is more like a general range from 40% to 80%. If it were only 40%, would you say the problem is not worth fixing? So what's the point of time accuracy, when what you really want is to find the problems?

  6. that counting of statement or function invocations is useful.
    Suppose you know a function has been called 1000 times. Can you tell from that what fraction of time it costs? You also need to know how long it takes to run, on average, multiply it by the count, and divide by the total time. The average invocation time could vary from nanoseconds to seconds, so the count alone doesn't tell much. If there are stack samples, the cost of a routine or of any statement is just the fraction of samples it is on. That fraction of time is what could in principle be saved overall if the routine or statement could be made to take no time, so that is what has the most direct relationship to performance.

  7. that samples need not be taken when blocked
    The reasons for this myth are twofold: 1) that PC sampling is meaningless when the program is waiting, and 2) the preoccupation with accuracy of timing. However, for (1) the program may very well be waiting for something that it asked for, such as file I/O, which you need to know, and which stack samples reveal. (Obviously you want to exclude samples while waiting for user input.) For (2) if the program is waiting simply because of competition with other processes, that presumably happens in a fairly random way while it's running. So while the program may be taking longer, that will not have a large effect on the statistic that matters, the percentage of time that statements are on the stack.

  8. that "self time" matters
    Self time only makes sense if you are measuring at the function level, not line level, and you think you need help in discerning if the function time goes into purely local computation versus in called routines. If summarizing at the line level, a line represents self time if it is at the end of the stack, otherwise it represents inclusive time. Either way, what it costs is the percentage of stack samples it is on, so that locates it for you in either case.

  9. that samples have to be taken at high frequency
    This comes from the idea that a performance problem may be fast-acting, and that samples have to be frequent in order to hit it. But, if the problem is costing, 20%, say, out of a total running time of 10sec (or whatever), then each sample in that total time will have a 20% chance of hitting it, no matter if the problem occurs in a single piece like this
    .....XXXXXXXX...........................
    .^.^.^.^.^.^.^.^.^.^.^.^.^.^.^.^.^.^.^.^ (20 samples, 4 hits)
    or in many small pieces like this
    X...X...X.X..X.........X.....X....X.....
    .^.^.^.^.^.^.^.^.^.^.^.^.^.^.^.^.^.^.^.^ (20 samples, 3 hits)
    Either way, the number of hits will average about 1 in 5, no matter how many samples are taken, or how few. (Average = 20 * 0.2 = 4. Standard deviation = +/- sqrt(20 * 0.2 * 0.8) = 1.8.)

  10. that you are trying to find the bottleneck
    as if there were only one. What you are looking for is activities that take a significant percent of time that you can replace with something faster, and there can be several of those. They are like acrobats forming a human tower, with the biggest guy on the bottom, up to the smallest on top. The biggest guy has the largest percentage of the weight, so that's the one to remove. Once you've done that, the next biggest guy has the largest percentage of the weight, so you can go after him next. That's the "magnification effect". Each problem you remove reduces the overall time and makes the others take a larger percentage of what remains. (Example: Fix a 20% problem. Time shrinks to 4/5 of what it was. Percentages of remaining problems expand by 5/4 to get back to 100%.) Since you're always removing, if not the biggest one, one of the big ones, that makes the remainder bigger, percentage-wise, so you can keep going until you can't anymore, and then you've made massive speedup factors.

** With the exception of other ways of requesting work to be done, that don't leave a trace telling why, such as by message posting.

Mike Dunlavey
Mike, have you thought about modifying valgrind to offer your stack profiles? I'd love to try this out in class.
Norman Ramsey
@Norman: I made a profiler based on this, in C for DOS, around '93. I called it yet-another-performance-analyzer, and demo'ed it around at IEEE meetings, but that's as far as it went. There is a product from RotateRight called Zoom that's not too far off. On *nix, __pstack__ is good for doing it manually. My to-do list for work (pharmacometrics on Windows) is about a mile long which precludes fun projects, not to mention family. This might be useful: http://stackoverflow.com/questions/1777669/how-to-modify-a-c-program-so-that-gprof-can-profile-it/1931042#1931042
Mike Dunlavey
... Here's a really short explanation on SO. Unfortunately, when you follow the link to it, if you don't see it, you then have to go to the bottom of the previous page of answers. http://stackoverflow.com/questions/406760/whats-your-most-controversial-programming-opinion/1562802#1562802
Mike Dunlavey
I've always found profilers not so useful for fixing slow code, and instead used selective bits of debugging code to measure the time taken by a group of statements of my choosing, often aided by some trivial little macros or whatever. It's never taken me too long to find the culprit, but I've always been embarrassed of my "bear skins and stone knives" approach when "everyone else" (as far as I know) uses the fancy tools.Thank you for showing me why I could never get the information I needed from profiler. This is one of the most important ideas I've seen on SO. Well done!
Wayne Conrad
@Wayne-Conrad: Thx. Actually there are profilers that sample the call stack, on wall-clock time, suppressing sampling during user-wait, and that summarize by % at the line / instruction level, such as RotateRight/Zoom. Oprofile can be made to do that too. Personally, I just rely on the pause button, because sometimes even the call stack is insufficient state information to say **why** something is happening.
Mike Dunlavey
Hey, please, don't rip gprof. Gprof is one of the most portable profilers and it works on platforms, where there is almost (or totally) no any other way of profiling programms.
osgx
Mike Dunlavey
Can you expand on point 3 a bit?
Andrew Grimm
@Andrew: Sure. Forget the difficulty of putting times on a graph, especially with recursion (point 4). Forget that call graphs are function (not line) based (point 2). Forget that call graphs are about time, not location (point 2). The need is to locate the lines of code that call for time to be spent for not-really-necessary reasons. So if a stack sample shows a series of lines of code, the reason for that nanosecond being spent can be "read off" the sample. Then if the reason is one that isn't really needed ...
Mike Dunlavey
@Andrew: ... *and* if that reason applies to some significant fraction of samples (like more than 1), then the line(s) of code that could eliminate that activity are on those samples. A graph can give you a *hint* of this, but a not-large number of stack samples will simply show them to you.
Mike Dunlavey
@Andrew: I'm sorry to belabor, but this effect is unbelievably pronounced in the big C# app I work in, with around 10^6 lines of code. No-one has ever made the graph, it would look like a rat's nest. But the call stack tends to be really deep (like 30) so when we want to speed things up, the hunting is really good in all those calls.
Mike Dunlavey
@Andrew: If you want to get academic about it, the argument runs two ways. If there is a problem of a certain fractional size F, then out of N stackshots, the number n that expose it is a simple binomial distribution. In the other direction, if n/N stackshots show a problem, then its size F follows a beta distribution.
Mike Dunlavey
This answer is well and good, but it's not a gprof alternative. Can you suggest something that *does* do stack sampling (and doesn't cost anything)?
Matt Joiner
@Matt: Well, pausing to get stackshots under a debugger doesn't cost anything, and it finds the problems, and gprof doesn't, so to me at least, that's an alternative. Then again, there's Zoom and LTProf (for money).
Mike Dunlavey
@Mike Dunlavey: I find that that breaking under a debugger still won't pause while the process is executing in the kernel, for instance waiting for IO to complete. That's the only thing gprof _doesn't_ do. Breaking and observing manually is just the same as gprof without the instrumentation.
Matt Joiner
@Matt: The debuggers I use (VC and VStudio) will break during IO and show me the stack down to the IO call, which is exactly what I need. Not sure about GDB, but even if it doesn't break until the IO completes, that's good enough because then the stack shows you the IO call in which the pause occurred, and the stack leading to it. If you get many of those, that tells you you're spending a good percent of time in that IO, and you can see from the stack if you need to. The difference from gprof (besides not capturing the stack) is (like in the visual studio profiler) ...
Mike Dunlavey
@Matt: ... that if you hit that key during a blocking call, you *find out* about it. You find out that you hit it during the blocking call, and you find out why it was there. In gprof and the VS profiler, you don't find out, because any sample during blocking is discarded, so it is totally blind to IO, and at some point during optimization, IO may well be the dominant performance problem.
Mike Dunlavey
@Matt: Examples of IO performance problems found this way: 1) printing log messages to a file or the console, which was erroneously thought to be insignificant. 2) Converting between text and doubles in numeric IO. 3) Subterranean IO extracting internationalized strings during startup, strings that it turns out did not need to be internationalized. I've hit lots of examples like these.
Mike Dunlavey
@Matt: Example: consider a program that loops 10,000 times. Inside the loop, it crunches numbers for 1ms (total 10s), and then does blocking IO for 9ms (total 90s), so 90% of its time is blocked. gprof and VS profiler will ignore the 90% and act as if the 10% was all there was to tell you about. Stackshots will show you that 90% of the time it is blocked, and why, and that 10% of the time it was crunching, and why. Of course, if you want it to go much faster, which is the part you need to work on?
Mike Dunlavey
@Mike Dunlavey: I agree with you in principle, I'm just not sure all debuggers are up to it. When using MSVC I do exactly the same thing. Your separation of IO (kernel) and actual CPU profiling are correct (I'm writing a filesystem atm, so naturally this is all very close to home and familiar). OProfile and Zoom are reasonable, but not brilliant. Callgrind is vaguely useful too (but has too much overhead), although it works very similarly to the debugger model you've given.
Matt Joiner
@Matt: It may be that not all debuggers are up to it, especially if the code you're trying to optimize is inside the kernel. Then you may need bigger guns to get the stackshots. For me, that meant using an in-circuit-emulator box. Also, maybe a software emulator can do it like Valgrind, I don't know. It may not be easy, but there usually is a way to do it. Not to brag too much, but so many times I've just had to go in and do it, while people are standing around guessing.
Mike Dunlavey
Good optimization skills are pretty rare. Many programmers never do it, don't need to do it, don't know how to do it or do it too early.
Matt Joiner
@Matt: Rare, yes, sadly, because it's so easy. And you don't need to do it until you do, and then you're stuck. SO is filled with questions and answers about how to guess at performance problems, when finding out is so simple.
Mike Dunlavey
@Mike Dunlavey: I think because there's a real lack of quality benchmarking tools. I'm not one to _rely_ on tools Windows-style (gotta catch 'em all), but a few good strong utilities.. gcc, gdb, make, gedit... gprof. No corresponding tool of choice exists when it comes to profilers.
Matt Joiner
@Matt: I'm sure I'm old-fashioned, but tools don't excite me unless they do. For timing, a stopwatch is all I've really needed, though it's nice to have time printed out. For profiling, MS IDEs are totally adequate if they have a pause button. I don't like GDB but it also works. There's no performance problem I can't find, quickly, with these. I'm amazed anybody uses gprof, for all the reasons above.
Mike Dunlavey
For some arguments against this approach, see [this question by Mike Dunlavey](http://stackoverflow.com/questions/266373/one-could-use-a-profiler-but-why-not-just-halt-the-program)
Andrew Grimm
@Andrew: It's charitable to call them arguments. Careful thinking is in short supply.
Mike Dunlavey
+12  A: 

valgrind has an instruction-count profiler with a very nice visualizer called kcachegrind. As Mike Dunlavey recommends, valgrind counts the fraction of instructions for which a procedure is live on the stack, although I'm sorry to say it appears to become confused in the presence of mutual recursion. But the visualizer is very nice and light-years ahead of gprof.

Norman Ramsey
@Norman: ++ That confusion about recursion seems endemic to systems that have the concept of propogating times among nodes in a graph. Also I think wall-clock time is generally more useful than CPU instruction times, and code lines (call instructions) are more useful than procedures. If stack samples at random wall clock times are taken, then the fractional cost of a line (or procedure, or any other description you can make) is simply estimated by the fraction of samples that exhibit it.
Mike Dunlavey
... I'm emphasizing call instructions, but it applies to any instructions. If one has an honest-to-goodness hotspot bottleneck, such as a bubble sort of a large array of numbers, then the compare/jump/swap/increment instructions of the inner loop will be at the top/bottom of nearly every stack sample. But (especially as software gets big and hardly any routine has much "self" time) many problems actually are call instructions, requesting work that, when it is clear how much it costs, doesn't *really* have to be done.
Mike Dunlavey
... Check this out. I think they are nearly on the right track: http://www.rotateright.com/zoom.html
Mike Dunlavey
A: 

Take a look at Sysprof.

Your distribution may have it already.

Søren Sandmann
sysprof generated pretty useless output, and difficult to read
Matt Joiner