Using simple-build-tool for benchmarks

Question

I'm trying to get sbt to compile and build some benchmarks. I've told it to add the benchmarks to the test path so they're recompiled along with tests, but I can't figure out how to write an action to let me actually run them. Is it possible to invoke classes from the Project definition class, or even just from the command line?

Answer 1

Couldn't you simply write the benchmarks as tests, so they will be run when you call 'test' in SBT?

You could also run a specific test with 'test-only', or run a main with 'run' or 'exec' (see http://code.google.com/p/simple-build-tool/wiki/RunningSbt for details).

Answer 2

Yes, it is.

If you'd like to run them in the same VM the SBT is run in, then write a custom task similar to the following in your project definition file:

  lazy val benchmark = task {
    // code to run benchmarks
    None // Some("will return an error message")
  }

Typing benchmark in SBT console will run the task above. To actually run the benchmarks, or, for that matter, any other class you've compiled, you can reuse some of the existing infrastructure of SBT, namely the method runTask which will create a task that runs something for you. It has the following signature:

 def runTask(mainClass: => Option[String], classpath: PathFinder, options: String*): Task

Simply add the following to your file:

  lazy val benchmark = task { args =>
    runTask(Some("whatever.your.mainclass.is"), testClasspath, args)
  }

When running benchmarks, it is sometimes recommended that you run them in a separate jvm invocation, to get more reliable results. SBT allows you to run separate processes by invoking a method ! on a string command. Say you have a command java -jar path-to-artifact.jar you want to run. Then:

"java -jar path-to-artifact.jar" !

runs the command in SBT. You want to put the snippet above in a separate task, same as earlier.

And don't forget to reload when you change your project definition.