Design of Assertion Checking System

Question

Hi All,

Coming to the point I am wondering if anybody could comment on the design of a "Performance Assertion Checking" system.

The idea is that a developer makes some assertions about his code and use these to test the evolution of the performance of the code.Has anyone has similar experience.

My current block is "Whats a better way to translate these assertions in a specified language (that are to be checked against specified logs or runtime instrumentation) into.. say CLR, or assembly or Bytecode that could be executed?"

Currently I have written a parser that parses the specification and holds it in a data structure.

Appreciate your time. Any Ideas..

Answer 1

Do we embed performance checks in our application? No. The reason is that the performance checks themselves take time and our application is very sensitive to performance.

Instead, we make our performance checks a test. And for that we use NUnit. For our nightly builds, we run the test, we generate a log with detailed timing data as well as a pass/fail indication given our requirements. Since we keep our logs around for some time -- forever for beta and production releases -- we can track performance over time as well.

Answer 2

Similarly to Kevin, I put performance logs in my automated regression testing, so that I am effectively regression testing performance as well as functionality. We use TestComplete for the automated regression and it does alot of this automatically. The principal reason for adding it manually is to compare results of this run versus last run at each checkpoint. This works something like

StartTest
InitialiseCounter
'
'
Do some testing
'
'
CheckPoint
GetElapsedTime
Compare ElapsedTime with stored elapsed time from last run
If difference is outside tolerence log an error

(Excuse the dodgy highlighting on my pseudocode)

Answer 3

Many languages nowadays have assert statements. Can they be leveraged to validate your generated assertion? They're easy to write and easy to find. The issue is that an assertion failure means your program stops.

If you want to provide a warning or a log entry that an assertion has failed at run-time, you could try an if-statement.

For this kind of code generation, people often use simple template tools to generate the appropriate source that can be inserted into the application. You could look at Java's Velocity or Python's Mako to generate source for your assertion condition.