What are the (dis)advantages of writing unit tests in a different language to the code?

Question

Unit tests have different requirements than production code. For example, unit tests may not have to be as performant as the production code.

Perhaps it sometimes makes sense to write your unit tests in a language that is better suited to writing unit tests? The specific example I have in mind is writing an application in C# but using IronRuby or IronPython to write the tests.

As I see it, using IronPython and IronRuby have several advantages over C# code as a testing language:

• Mocking can be simpler in dynamically typed languages
• IronPython has less verbose type annotations that are not needed in unit tests
• Experimental invocation of tests without recompilation by typing commands at the interpreter

What are the tradeoffs in using two different languages for tests and production code?

Answer 1

The biggest tradeoff would be if someone was looking at using Unit Tests to figure out how a certain action may be performed, writing it in a different language would make it harder to use. Also you would have to make your C# code CLS compliant.

Answer 2

Main disadvantage as I see it is maintainablity. If you code in C#, your development team are competent in that language, as will new hires be. You need a multi-functional dev team.

I think that it's also worth noting that you probably don't want people writing their tests in a language that is maybe not their strongest. Your test code needs to be robust.

You also need to be switching between syntaxes whilst writing codes/tests - this is a bit of a nuisance.

Answer 3

One obvious potential problem is a confusing debugging experience. I haven't personally tried to debug across languages in .NET - what happens if you step into C# code from IronPython?

The other problem is that anyone wanting to develop on your code base has to know both languages.

Answer 4

Disadvantages that come to my mind:

• Depending on the language, you need another development environment (additional dependency of the project, additional effort to setup a development machine, additional licenses, additional training ...)
• Refactoring is sometimes supported by the IDE - but most probably not for this other language. So you have to refactor them manually.
• Unit tests can also be used as programming examples. Tests show how the tested classes are intended to be used. This does not work so well if the tests are written in a different language.

Answer 5

When building an API or library, I often deliver unit tests as a form of documentation on how best to use the API or library. Most of the time I build a C# library, I'm delivering to a client who will be writing C# code to use the library.

For documentation sake, at least some of my tests will always be written in the target language.

Answer 6

The biggest disadvantage is that you are also testing compiler dependencies in your unit tests. In a sense, this also makes them integration tests. That might make them preferable if you expect your code to be usable from multiple languages, but it's adding one level of complexity that you may not need if your code will only be used in production with the language that it's developed in.

One advantage that I can see is that it further isolates that code being developed from the test itself. By separating the act of writing the test even further from the actual code under development, it forces you to really consider how the code should be written to pass the test rather than simply moving the initial development of the code into the test itself.

Answer 7

I'm a huge fan of the Iron languages but there are significant disadvantages in using them to test statically compiled code. Testing Ruby/Python with Ruby/Python works great, i.e., you can fake objects in any number of ways. But testing C# with Ruby/Python can lead to more complexity than may be desired.

Consider a scenario where you are writing an ASP MVC project where you would like to fake out the type of browser making a request to the controller. In C#, you could mock it out like this (using Moq):

var fakeRequest = new Moq.Mock<System.Web.HttpRequestBase>();
fakeRequest.Setup(request => request.Browser.Type).Returns("Fake");

In IronRuby, it would look something like this:

class FakeBrowser < HttpBrowserCapabilitiesBase
  def type
    "Fake"
  end
end

class FakeRequest < HttpRequestBase
  def browser
    FakeBrowser.new
  end
end

The deeper the context that needs to be faked out, the more complicated it gets. In C#, the mocking framework does the subtyping automatically for you but in a dynamic language you will, unfortunately, have a lot of work ahead of you. Mocking libraries like PyMock or Mocha will not work for tests like this because the code under test has strong typing dependencies and the fake objects generated by Ruby/Python frameworks will not conform to those dependencies. I would love to see the situation improve as both IronRuby and IronPython mature but right now the testing C# story isn't so good.

If my take on this is wrong, I'd love to stand corrected. =)

Answer 8

I think that this is an excellent question and an idea worthy of consideration - particularly in an environment like Visual Studio/.NET where this is easily supported

The plus side - as you suggest - is that you can choose to use a language/tool to create tests that is more suited to creating tests than perhaps the code you are using to create code and for this reason alone its worth a thought.

The down side is, as suggested, that your developers - those creating the tests (and we must remember not to confuse Unit Testing with Test Driven Design) probably ought to be fluent in more than one language (I'd suggest that the ability to be so is fairly important to a good developer but I'm biased!) - and more importantly that you may have to worry about structural differences between the two (though again, if you're talking about .NET languages that should be covered for you).

It gets even more interesting if you go beyond "unit" tests to tests at all levels where the specific capabilities of particular languages may give advantages in building up a test case.

The ultimate question is whether the advantages outweigh the disadvantages... and that's probably somewhat case specific.

Answer 9

I agree with the other answers that there are disadvantages, but I'm surprised that so few have talked about advantages.

• TDD'ing across languages is a great way to learn new languages: write the tests in a language you know well, and the implementation in the language you are learning. As you are learning, you will discover better ways of writing the code than you first did, but refactoring is easy because you have a test suite.
• Having to master multiple languages keeps you (and your team) sharp.
• You get better verification that your API is interoperable across languages.