Unit Testing Third Party ORM

Question

Hi,

I've read a few threads on SO about usefulness of unit-testing various applications. The opinions can range from "test everything all the time" to "unit tests are useless", and everything in between ("test where it makes sense"). I tend to lean towards the middle one.

That leads me to my question. I am trying to decide if it would be beneficial or practical to have some basic unit-tests testing 3rd party ORM as suggested in this SO post: link text

some baseline tests may be useful as insurance against future breaking changes, depending on how you are using the tool. For example, instead of mocking up the entire n-tier chain (I'm not a fan of mocking when it is not necessary), just use the ORM tool to create, read, update, and delete a typical object/record, and verify the operation using direct SQL statements on the (test) database. That way if the 3rd-party vendor later updates something that breaks the basic functionality you'll know about it, and new developers to your project can easily see how to use the ORM tool from the unit test examples.

My main reservations following this advise is that it would require way too much setup, would be a headache to maintain, and over all it would not be practical in our environment. Here's the summary of some points to consider:

1. The ORM we're using requires static datasource object(s) to be created and registered with its Data Access Layer and associated with authenticated user. This would require a lot of test setup, and probably would be problematic on the build server where no user is logged on.
2. ORM vendor has a pretty good track record of releasing new updates and not breaking basic functionality. Furthermore whenever it's time to update ORM to the latest version, I would imagine that application wouldn't go straight to production, but would be thoroughly regression tested anyway.
3. Maintaining test db for Unit testing is kind of problematic in this environment. Test db gets wiped out after each major release and replaced with db backup from staging with sensative data obfuscated. I would imagine in order to have a test db for ORM unit testing, we would need to run some scripts/code that would set the database in a "test" state. Again too much setup and maintenance.
4. And finally ORM documentation/help for new developers. I can see how something like that could be useful. But ORM vendor provides pretty good documentation/help with demo apps. So writing unit tests on top of that doesn't seem to be worth all the efforts.

So, is it worth to go through all these troubles just to make sure that ORM does what it supposed to do (which is CRUD)? Shouldn't it be a responsibility of the vendor anyway?

Answer 1

In this particular case, I wouldn't bother. I think you are correct in assuming a bad ROI here.

And yes, I consider it the responsibility of the vendor. I expect (and assume) their stuff works. That's how I treat my vendors.

Answer 2

You said it yourself. Test where it makes sense. If it will make you "feel" better to test the 3rd party ORM, then do it. But, ultimately, you're putting your trust in their tool. What are you going to do if the ORM suddenly stops working? Have you written enough code against it that you can't easily rip it out? You'd wait for them to fix it, probably.

Basically, you have to treat 3rd party tools as the proverbial black boxes and let them do what you bought them to do. That was the reason you paid the money you did, right? To keep from having to write that yourself.

Answer 3

It is the responsibility of the vendor to make sure the ORM does what it's supposed to do, but it's your responsibility to ensure that your application does what it's supposed to do, and if it fails for whatever reason, your clients will be unhappy, even if it's "just" because the ORM failed.

Your tests could ensure that the ORM works the way you expect it to given the way you're calling it. It's possible that the ORM will change in a way that isn't "broken" but that doesn't play nicely with your application.

That being said, if you're confident in the ORM, and feel that setting up and maintaining any kind of automated tests of the ORM is not worth the effort, it's probably not, especially if you've got other levels of testing that are likely to reveal the problems if they arise.

Answer 4

I personally think that real unit tests should only test the application itself, and everything that needs to be separately deployed and configured should be mocked up.

What you are saying is to write some integration/functional tests, that test the whole system end-to-end. These will never be lightweight, but probably are still useful in some cases (e.g. if your system doesn't change too much and is critical for your company at the same time). I have seen such tests automated as well, using virtual servers (either VMWare or microsoft equivalent), and an example database which was restored from file before every test run. You can also just set the ORM once, and accept that the tests will fail mainly because the configuration will break. Obviously you can test more, but be aware that the cost is higher.