views:

308

answers:

3

Hi all,

I have stumbled on an impossibly good performance behaviour with PostSharp. To evaluate the speed I wrote a little program, that would execute one function a specified number of times, and if PostSharp is enable it would generate and delete a few hundred strings, just in memory (non fixed composition, so they are not auto-interned). The loop executes in a non-trivial (a few milliseconds) amount of time.

Now, I am unable to measure the difference on a few million runs, and a crazy run of ~40 billion iterations amounted to a difference of just a few nanoseconds vs non-PostSharp version doing the same number of calls. To me, this is impossible. There must be something wrong with my test. I had the code peer-reviewed by my co-workers, so I am fairly confident the code does what I intend it to.

So, is there something wrong with using string generation (which is the expected use in the intended applications) as the slow-running simulation for the benchmarks?

Alternatively, has someone else performed (or know of) a PostSharp's runtime performance analysis?

Thank you.

+1  A: 

On a 3 GHz processor, 40 billion clock cycles alone will take 13 seconds - and I sincerely doubt that a single iteration is taking just one clock cycle. Something's definitely wrong with your test.

Something's likely getting optimized away - maybe it sees that you're doing the same thing over and over again and is deciding not to do it at all (except the first time). You need to make sure you're randomizing your data when you do perf analysis.

Aaronaught
Like I said, strings are not fixed. They include random numbers. Furthermore Debug builds in VS don't optimize anything away. They are very dumb this way, which is why they are a good reference for consistency.
Alex K
@Alex: Maybe you should include a skeleton of your testing code. Without specifics, I think nobugz and I have said as much as can be said. A running time of a few ms for 40 billion iterations is impossible, as is a delta of only a few ns between tests.
Aaronaught
One more thing - how long does it take to run the entire (n = 40 billion) test? Doing *anything* that many times is likely to take at least several minutes to run; if it's happening any faster than that, then it's probably not running that many times.
Aaronaught
Takes a little over an hour. As for code, there really not much to sample. There is a for loop calling the function, a for loop in the function generating random strings (to make sure the function is not instantaneous and doesn't get optimized away in Release builds) and a for loop in the postsharp attribute generating random strings.
Alex K
@Alex: You mention that if the PostSharp attribute is present, it generates and deletes a few hundred strings. Over 40 billion iterations, this alone would take more than an hour. My psychic debugger tells me that the PostSharp code is never getting executed, or perhaps is getting executed outside the test loop. Have you set a breakpoint in there?
Aaronaught
Hmm... I did set a break point there when I was first setting it up/learning, but before I added loops. I didn't try to see if the loops get executed consistently. Perhaps you are right, and at least something gets optimized away. I shall try it tomorrow. Thanks for the feedback.
Alex K
@Aaronaught: Alright, problem found. I have a base class in common projects with functions defined and a derived class (no function override) inside other projects (I have one project per configuration). The derived class has a post sharp attribute, the base class doesn't. If I call methods on the derived class that it inherits from the base class, PostSharp attribute is not triggered for methods with parameters, but is triggered for parameterless ones (but only once). I dunno if that is how it's meant to work (I think I still have 1.5 beta) I've updated to final + fixed code structure.
Alex K
Just to clarify, attributes have only been applied to the class, not functions. If the function is not implemented in the class, the attribute does not trigger. If the function is defined in the class, the attribute triggers.
Alex K
Interesting to know. Attribute inheritance often seems to lead to these subtle bugs - there was another question a few days ago about someone having similar issues with XML attributes. Most likely the PostSharp code is inconsistent in the way it checks for inherited attributes (i.e. sometimes passes `true` to `GetCustomAttributes`, and sometimes `false`), but one would have to scour the source code to be sure.
Aaronaught
+1  A: 

I have done performance tests. They were published in http://www.postsharp.org/blog/introducing-postsharp-20-2-amazing-runtime-performance-enhancements.

Some aspects can have the same performance as hand written code if they don't use features such as: reflection, access to method parameters, access to method instance. Since PostSharp emits MSIL instructions, the generated code can be inlined by the JIT compiler.

As reminded in other answers, be sure that (1) PostSharp is indeed invoked (use Reflector on the resulting assembly) and (2) you're using the Stopwatch properly. If you're comparing the average time of a single test, it's normal that the difference between PostSharp and hand-written code is just a few nanoseconds (in the hypothesis that you don't use expensive features).

-gael

Gael Fraiteur
A: 

Can you change your test, such that the generated strings are used in the next iteration (string length written to the console) or something like that? Maybe the compiler optimizes your program in such a way that either the postsharp-function is not executed at all or that it is called asynchronously or executed on another cpu, because there is no reason to sync with the other iterations. If you link it more tightly, this may force then the compiler, to synchronize the actions.

Simpzon