views:

1066

answers:

14

This could be a borderline advertisement, not to mention subjective, but the question is an honest one. For the last two months, I've been developing a new open source profiler for .NET called SlimTune Profiler (http://code.google.com/p/slimtune/).

It's a relatively new effort, but when I looked at the range of profilers available I was not terribly impressed. I've done some initial work based on existing products, but I felt this would be a good place to ask: what exactly do you WANT from a profiler?

I come from real time graphics and games, so it's important to me that a profiler be as fast as possible. Otherwise, the game becomes unplayable and profiling an unplayable slow game tends not to be very enlightening. I'm willing to sacrifice some accuracy as a result. I don't even care about exceptions. But I'm not very familiar with what developers for other types of applications are interested in. Are there any make or break features for you? Where do existing tools fall down?

Again, I apologize if this is just way off base for StackOverflow, but it's always been an incredibly useful resource to me and there's a very wide range of developers here.

+3  A: 

If it did the same things that JetBrains dotTrace does, I would be very happy indeed.

Robert Harvey
+13  A: 

My requirements:

  • Collect stats without impact to application - e.g. don't fill up memory, allow data to be collected away from apps under question
  • Ability to specify measurements simply and repeatably (data driven)
  • Automatable so that I can repeat measurements without point and click, and without UI
  • Enable us to understand issues related to the WPF and other declarative technologies such as DLR or WF
  • No installation - no gac, msi etc, even better if can be run over a network
  • Support 64 bit from outset
  • Don't try to know all the the analysis that could be done - encourage an ecosystem. If the raw stats can be analysed using other tools so much the better.
  • UI if any should be good - but its the stats that matter. So don't spend time on that, get the core profiling good.
    • Support profiling of apps that are not straight exe's like services and web applications simply.

would likes:

  • Consider cross app support - big apps often need to understand apps performance behavior across many executables. If your profiler allows easy correlation of this data then so much the better
Preet Sangha
I actually hit some of these already -- data is broadcast out of the profile target via socket, to either local or remote frontends. The backing datastore is SQL Server Compact, with more options coming eventually. I will look at doing a robust automation system, as I agree that it's an extremely important feature that is missing in many existing tools, and data driven will tie into that too.
Promit
+1 for 64Bit from outset
TWith2Sugars
+1 for 64Bit from outset
Meidan Alon
you say the backing store if CE - would this need to be installed?
Preet Sangha
+4  A: 

There is EQATEC Profiler that's a free .Net profiler that I have been meaning to use.

One thing I would like to see is Mono compatibility. I have started dabbling in Mono, and it would be great to have a .Net and Mono profiler!

phsr
+9  A: 

My wish list:

  • Really easy to use - simple (yet powerful) GUI
  • Spectacular performance - ability to profile apps that are under extremely heavy usage.
  • X64 and X32 support
  • Understands SQL, is able to give me stack traces and duration for all my SQL calls, coupled with SQL.
  • Easy to profile, no need to go through a complex, recompile the app process.
  • Easy to profile services, web sites and processes which are launched as side effects
  • A "production mode" which allows you to gather key stats from a production based system.
    • "Automatic bottleneck finder" : run against a production app and using heuristics determine which methods are slow.
  • Per thread analysis, tell me which threads are doing all the work and where.
  • Profile at various granularities, allow to perform a "cheap" profile that only gathers key info and dig in with granular profiling.
  • Exception tracker, allow me to track all the exceptions which are thrown in my app (key stats and detailed info)
  • Per thread profiling - allow me to profile a single thread in an app
Sam Saffron
+4  A: 
280Z28
Ooh, pretty. I don't know if I'll implement line-by-line any time soon -- it seems like a low yield high cost feature to me. But the top view is gorgeous and I'm definitely taking ideas from that. Time to put my MSDN subscription to good use I suppose.
Promit
That's a nice screenshot. For me line by line is quite important as even something simple can take a lot of time if its called enough so it's nice to know exactly where the time is spent.
Ian
Well, line by line is too expensive to use in general, but it *would* be nice to be able to enable it for specific functions. I find it annoying that if I want to find out which line is slow in a function, I need to abuse "extract method" a lot.
Brian
@Brian: Something tells me you haven't used the VS10 profiler. That something would be claiming it's slow. At least for sampling mode, it's not. :D
280Z28
I top that VS 10 or 2010 has a very good profiler.
Jonathan Shepherd
+2  A: 

I'll add one more here that would really be sweet. Make a simple assembly that has a Mark(string) function available, where if the application called that method, then in the results you could select to see the results just from there to (the end | some other specified mark). Another possibility is BeginSequence and EndSequence or something. Double plus if you can specify whether the mark applies only to the current thread, or to all threads.

280Z28
Since I'm from games, this one is super-high on the list -- profiling frame-to-frame is absolutely crucial.
Promit
A: 

It would be nice if the .NET related profiling measures from Perfmon are integrated, so that you avoid "double" monitoring with perfmon and your app. This is esepcially useful for all the memory-related items.

weismat
A: 

My prefered profiler was "DevPartner Performance Analysis Community Edition" (http://researchlibrary.theserverside.net/detail/RES/1165507815%5F474.html?psrc=MPR), unfortunately it's no longer available.

What made it stand out against the competition was the graphical analysis that showed a box for the current selected method and outgoing connectors to called methods showing the percentage of time spend in each. Also connectors to incomming calls. Of course thos caling and called methods had the same and you could expand them as needed This way, you could navigate freely along your call stack, see the stack as deep as you want, and tackle the hot path in your fragment.

The second demand would be "ease of use", i.e. it should run with all relevant application types, windows exe, web app, windows service, WCF service, (Silverlight?), ... . And not only with tiny sample applications, but with enterprise sized not-so-trivial applications.

+1  A: 

I'd like at least some compatibility with ASP.NET, though I understand it's actually pretty hard to get that to work.

Line-by-line is so nice in Shark that I'd also like to have it in .NET.

A choice of visualizers is a good thing - I'd like to see a bunch of different call trees, statistical graphs and maybe even a heat map of which methods are called most frequently.

ravuya
+2  A: 

What I would like on a profiler:

  • Should work on 32 and 64 bits
  • Should have componentes for all tiers (client, application server, database) and some way to correlate between them. For instance, it would be nice to see how changes made to any of the tiers impact other tiers. That could help deciding on which tier to implement specific features.
  • Command line interface for using with automated scenarios (build server, stress testing, etc)
  • Should have a lightweight sampling mode and a more precise instrumented mode. The second one should impact execution measurements as less as possible.
  • A GUI for ease of use and that should generate the necessary config files for using em comand line mode
  • Generate results in a starndard format (if such thing exists) so I can consume the results with other tools
  • Should also generate or export results to Visual Studio's format (*.vsp)
  • Compare between two or more result files, to see the evolution or regression of the code base.
  • Collect and correlate the target application data with PerfMon data of other processes running on each target machine to identify concurrent resource usage (Memory, Processor, Disk and Network I/O)
  • Determine thresholds for which some alert mechanism should be invoked. Example of such would be to e-mail someone if a specific scenario takes more time than specified.
  • Hability to attach and detach from a running process to collect sampling data without interfering with the target application. Must have for using on production sites.
Alfred Myers
+1  A: 

A couple things I'd really love to see:

Data Collection:

  • An option to allow tracking of context through new thread. That is, when a call to either new Thread() or ThreadPool.Queue...() happens, count the work performed by the other thread as though it happens inside the calling function, even though they occur on different threads, and the calling method isn't actually blocking. This would ultimately allow one to identify code that produces a lot of work in a common method that implements an async pattern. This really could be great!
  • Tracking allocations inside methods. There's a chance the .Net memory profiler already does this, but identifying which methods perform many allocations could be invaluable. Even if other tools can do this, having everything in one tool is always great.
  • An aggregation capable of detecting "spikes" in usage and analyzing only them. This could be handy when analyzing a background process that misbehaves unexpectedly and infrequently.

UI end:

  • The ability to compare two runs, and highlight major differences between them.
  • Call Tree navigation and hot-path expansion (VS-style) would be great too.
sirob
+1  A: 

Phsr already mentioned the EQATEC Profiler .

One feature it has that I like is that, even without reading any documentation or paying any attention at all to what I am doing, I was able to successfully, from start to finish, profile an application. Usability is a wonderful thing. Please be careful about how you add all those fancy options...don't kill usability in the process.

Brian
+2  A: 

Years ago I built a profiler, and described it on SO in answer to some other question I can't locate right now, about how to build a profiler.

It is based on at least partially automating the technique I have used for decades, of which an example is given here. It is based on stack sampling, and the key is in how that information is presented, and the thought process that the user goes through.

The general beliefs about performance tuning, that are taught in school (by professors with little exposure to real-world software) and continued via the 50,000-programmers-can't-be-wrong phenomenon, I suggest need to be questioned and put on a firmer footing. I am far from alone in feeling this way, as you might gather from cruising around SO.

I think profiler technology is gradually evolving (for the better in my opinion) toward stack-sampling and ways to explore the results. Here are the insights I depend on (which you might find a little jarring):

  • Uncovering performance problems so they can be fixed, and measuring performance, are two entirely different tasks. They are means and ends, and should not be confused.

  • To uncover performance problems, what is needed is to find what activities account for large amounts of wall-clock time being spent and that can be replaced with something faster.

  • The nice thing about such activities is, the very fact that they take time exposes them to random-time samples of the state of the program.

  • Not many samples are needed, if they are taken during the interval you care about. I.e. there's no point in taking samples while waiting for user input. To that end, in my profiler, I let the user trigger the samples with keys.

  • The reason you don't need many samples is this. Any given performance problem costs some fraction X of the wall-clock time in the interval of interest. A random sample in that interval has an X probability of "catching it in the act", so if N samples are taken, the expected number of samples catching it in the act are NX. The standard deviation of that number of samples is sqrt(NX(1-X)). Example, if N = 20, and X = 20%, you can expect roughly 2 to 6 samples to show the problem. That gives you an imprecise measure of the problem, but it does tell you it is worth fixing, and it gives you a very precise location, without any further detective work.

  • Problems typically manifest as more function, procedure, or method calls than necessary, especially as the software gets large, with more layers of abstraction, and thus more stack layers. The first thing I look for is call sites (not functions, but call statements or instructions) that appear on multiple stack samples. The more stack samples they appear on, the more they cost. The second thing I look for is "could they be replaced?" If they absolutely cannot be replaced with something faster then they are simply necessary and I need to look elsewhere. But often they can be replaced, and I get a nice speedup. So I'm looking carefully at particular stack samples, not aggregating them into measurements.

  • Recursion is not an issue because the principle that the cost of an instruction is the percentage of time it is on the stack is the same, even if it calls itself.

  • This is not something I do once, but in successive passes. Each problem I fix makes the program take less time. That means that other problems become larger fractions of the time, making them easier to find. This effect compounds, so that dramatic cumulative performance improvements are often possible.

I could go on, but I wish you luck, because I think there is a need for better profiling tools, and you have a good chance.

Mike Dunlavey
+1  A: 

One of the things I mis in almost all profiles is a managed API to perform automated profiling and automated tests.

I can imagine that you think, WTF... why would one like to automate profiling?

The answer is that some of our customers have requirements regarding speed, mem usage, etc. So for every new release we like to run a test on the stuff mentioned before shipping it.

Henri