Where do you keep your test data files?

Question

Sort of a spinoff of this question. Do you keep them in the source tree? Do you keep them in source control?

I'm thinking that if your test cases refer to files, then the files are part of the behavior specification of the system, therefore they're associated with the current version of the system, therefore they should be checked into source control. But I don't think they should be checked out locally, because they don't need to be, and they can potentially be quite large. So I'm leaning towards having a parallel tree, such that if a project's code files are at $svn/Code/foo/bar/baz, the associated test data files are in $svn/TestData/foo/bar/baz, and the latter will be accessed directly from the server using some kind of common test data helper class (which maybe caches files locally?), which can just accept relative paths and figure out where to find them. Does this make sense?

I guess there's the related question of how extensively I should be using external files for testing in the first place. I do think they're often good for higher-level "acceptance" tests.

Answer 1

I add them to source control in this way

- Trunk
     - Source
     - Lib
     - Tests
         - DescriptiveTestName_1
         - DescriptiveTestName_2
- Branches
     - v0.9
         - Source
         - Lib
         - Tests

This way all files source, libraries and tests are together for each version and it's easy to get up to speed when developing (and fixing issues). Each subdirectory under tests contains all the required files for the test.

Answer 2

We subscribe to the "everything goes under source control" school of thought (anal retentive doesn't even begin to describe us).

That means test cases (code and data) for all levels of testing that we're responsible for (unit, system, integration, translation ...), development software CD/DVD images, OS images, VM's for the test environments, all doco, basically everything that would be needed to put together the development/test environments if all PCs and software in the team were to be stolen - except for the hardware but that's only because we haven't found a way to check it in yet :-).

Disk space is far cheaper than the time it would take to re-source everything we need. And when a version of the software is released to the wild, we actually check out everything needed to construct that version, burn it to multiple DVDs and test the process on virgin hardware. Then we make multiple copies and distribute it to the four corners of the universe ... oops, sorry, got carried away.

But we do do everything I stated although the build DVDs are distributed to multiple geographically separated locations (on the planet, not across the universe).

As to where it goes in the source control tree, the top level of our tree is always a version, everything to do with that version exists under there. This gives massive duplication but makes things very easy to manage. And, generally, disk storage is a lot cheaper than human resources.

Answer 3

But I don't think they should be checked out locally, because they don't need to be

Why not? Tests are an integral part of any complex software system. If they can't run without their data, then they become useless.

The idea of remotely grabbing test data as it is needed is intriguing, but then you become reliant on a connection to your Subversion server when all you need to do is run tests. I think it adds unnecessary complexity to what should be a simple thing to do; running tests should never be a bottleneck to development.

In addition to this, you might want to consider the fact that you have to now upkeep two different svn trees. This could become a nightmare when you have multiple feature branches and releases or tags.

To explicitly answer your question, we keep our test files in <project root>/tests, so that each branch has its own set of working and useful tests.

Answer 4

I definitely think the test data should be versioned, in the same tree as the tests and source, so that it is easy to pull down the latest code and run the tests. I set up my trees like this:

trunk/
  +- Source/
  +- TestSource/
  \- TestData/

The tests then refer to ../TestData/myTestData.xml or whatever they need.