? Structuring a Revision Control System (SVN) to Handle Dependencies

Question

Hi,

For many years, I have been programming in a simple way: I would save my source files in directories organized by language and project, make the occasional manual backup, and if I’m smart, I make a copy before trying out a new version; that’s pretty much it.

I recently decided to begin using revision control. After examining a bunch of articles and pages, and trying out quite a few different ones, I eventually settled on Subversion (even though it doubles the size of the project due to the BASE).

I now need some advice on a few aspects that I cannot seem to find useful information on. First I am checking that I have the basic process of using an RCS correct:

1. import all my projects into SVN repositories
2. delete the originals
3. check out a project from the repository
4. work on it
5. commit it

That’s it? So what about new projects? Do I have to create a new project in a folder then import it?

I’m also having trouble with the directory structure, but first, I should probably lay out my setup. I am a single developer working on my home machine. I have in my Data directory the following sort of layout:

  X:\Data
      \H
        \3rdParty
          \Graphics
        \Controls
          \ThisControl
          \ThatControl
        \Libraries
        \Classes
          \CFoo
          \CBar
      \VC
        \Big
          \CoolApp
            \res
        \Small
          \CoolerApp
            \res
            \misc
        \Test
          \CFooTest

…and so on.

I have several of the header directories that I use often (eg 3rdParty\Graphics, Classes\CFoo, etc.) in the Include path of my IDE. The dependencies were already problematic before, but now with RCS, it’s even worse. For example, CoolApp may include ThisControl and CFoo. Previously, this would have been less than ideal since if I modified CFoo while working on CoolApp and broke it, other apps that use it like CoolerApp would thus be broken as well.

The reason I did it this way instead of copying CFoo et. al. to CoolApp’s and the others’ directories is because of the hassle of trying to merge the updates from each copy back to the main copy in the \H folder.

I would have thought that by using a formal RCS, this sort of problem would be obviated. However, what has now happened is that when I import the projects from \VC\CoolApp, etc. into the SVN repositories, the components like CFoo, Libraries\* etc. do not get included since they are in an external directory, and thus are not versioned—thus defeating the whole point.

I am looking for tips on how to deal with this sort of situation. For example if I have CWidget in \H, and WidgetTest (a test container that includes CWidget) in \VC, then how would I structure things such that both WidgetTest and CWidget get versioned, while at the same simplifying it as much as possible for other apps that use CWidget to include and use the latest version of it?

Also, I have only been able to import all of my projects into the same repository directory, losing the *Big\*, *Small\*, *Test\*, etc. structure. I can’t get Subversion to keep that.

Finally, what becomes of the original project directories? I’ve seen at least one article that says that they can be deleted once they are imported to the repository. If so, I’ll probably just ZIP them up and put them away.

Oh, and I currently have Subversion set up with my Apache server, as well as VisualSVN, SVNServe, and CollabNet SVN Server installed. I have gotten each one to work, but would love some advice on which to use since I’m sure I only need one.

Thank you very much.

Answer 1

The subversion book, free online, gives several suggested repository configurations:

http://svnbook.red-bean.com/

This page gives good links on further reading:

http://svnbook.red-bean.com/en/1.5/svn.tour.importing.html#svn.tour.importing.layout

One very nice thing about SVN, which is not true for CVS, is that moving around directories is pretty trivial. So don't feel pressured to come up with a "final" organization right of the bat. Do something that works, play with structure till you like it more.

One other thing worth mentioning, SVN uses a copy-on-write technique for data. So feel free to make "svn cp" copies of entire directories whenever you need to.

Answer 2

Ok, I'm going to concentrate on the shared project part of your question, since I've just come from a workplace where we had multiple projects and multiple shared projects in subversion.

The first thing I'd suggest as being crucial is that you start working with the following idea: "Doing a checkout of the trunk of a project anywhere on the hard disk is all that's required to be able to build the solution"

Also remember that working copies with no changes have no value. It shouldn't matter where on disk you check things out and deleting a working copy shouldn't start you panicking as you can just checkout trunk again and start up straight away. I found at my previous places a great deal of care is taken "crafting" the development environment and getting everything in just the right place. This is absurd, and time spent doing this is never got back. Do it once, do it in a decent source control (like svn, or git, or TFS) and be happy that you can now through working copies around like yesterdays newspaper.

The only time a working copy has any value is if there are any modifications that haven't been committed. Any valuable working copy is vulnerable. Always get modifications committed as soon as is feasible. Any form of hard-drive failure (including working copy corruption), accidental deletions, changes from something that works to something that doesn't, etc will have the potential to loose a lot of your work - that's the valuable bit. If your half baked code isn't fit to be seen in the trunk of a project, make a branch and commit to that.

This means that you'll be able to checkout multiple versions of the same project in different locations on your drive and work on them (eg, development and live-bugfixing versions). It also means that checking out the trunk will bring down all dependant libraries underneath that working copy folder.

It also means that having IDE level definitions of where headers/libraries are located just won't work. This was a terrible idea that visual studio introduced, but they do also let you specify this stuff using relative folders in individual project settings - where it should be. Trust me, if you're using IDE level location definitions, at some point you're going to be building your app and trying to work out why your changes aren't appearing. Then you'll get that sinking feeling when you realise you've just built your last 3 releases against an old, buggy version of a library. At least I did.

To get to this utopian situation, you'll be a lot better off if you consider each project & library (eg, CoolApp, CFoo, ThisControl, CWidget) as separate projects with separate release cycles, trunks, et al. Move towards thinking of these things independantly, developing and releasing them separately.

This sounds like a lot of overhead, but if you want to make changes to a shared component without breaking other projects that use it - it's a necessity.

With that in mind, I'd suggest you structure your repository something like this:

/Projects
 /CoolApp
  /trunk
  /branches
  /tags
/Libraries
 /CFoo
  /trunk
  /branches
  /tags
 /CWidget
  /trunk
  /branches
  /tags
 /ThisControl
  /trunk
  /branches
  /tags 
/Vendor
 /NUnit
  /current
  /1.6
  /1.7

If your repository isn't currently setup like that, you can use svncopy to structure it. You could use a tags/trunk/branches at the top level and everything underneath, but in my view doing that makes conceptually separating the projects more difficult.

Now checkout just the trunk of your main project (CoolApp). Of course - this won't build as is, because none of the dependant projects are there. The next step is to add the other projects as externals. On the top level folder of your working copy using tortoisesvn right click and go to svn->properties. Add a new property named "svn:externals". Define properties in the format

/repository_location[@revision]    working_copy_folder

So for coolapp, you might add the following svn:externals definition:

/Libraries/CFoo/trunk        CFoo
/Libraries/CWidget/trunk     CWidget
/Libraries/ThisControl/trunk ThisControl

When that's done, commit the changes, then do an "Update" and your externals will be brought down as well.

You can build any working copy folder structure you like using this form of externals definition. You will have to modify your solution file to look for the projects in their new location, but this isn't usually a big problem.

Once you're up and working this way, take note that it's generally a very bad idea to have externals pointing to a trunk. This is because when the library's trunk changes because of another project, when you do a new checkout of your solution (or an update), suddenly your perfect solution won't build - even though you didn't think you changed anything in it. Start making tags for your libraries and point your externals at them instead. This is when you start thinking of your libraries as separate projects in their own right.

This is a situation for another principle: "Any changes to the project MUST have a corresponding revision in the trunk of the main project" Changing the externals location to point to a new library version is a perfect example of being able to say "Hey, I'm now using version 2 of foolib here". The problem in the previous paragraph was because the code changed "underneath" you.

To modify a library you'd ideally checkout a working copy of that library's trunk, modify it and tag it with a new version number. Then in the main project, change the external to point to the new tag and Update.

You can shortcut this process by "switching" your library location from the tag to trunk. That way, when you commit, you'll commit the library changes to trunk. Working this way, however, you'll need to remember that the external still points to the tag - you'll still need to tag your library up and change the project external or a new checkout won't build.

If your rely on any 3rd party tools, like nAnt or nUnit - add them to the repository as vendor branches and reference them through externals. VS allows you to reference dll's by browsing for them, which is more flexible than from the GAC. It might not sound like much of a problem for a single developer, but if you try to upgrade nUnit one project at a time, you'll see how much nicer it is to be able to checkout the trunk of your project and know that you've got the right version of nUnit along with it (rather than then having to uninstall nUnit and reinstall the right version).

Please also note that it's not necessary to have every project in your solution as a separate external. It's only the things that can or should be shared amongst different projects that should be separated out.

Finally, once you're up and running, I'd really suggest using a build engine, like CruiseControl.Net or Hudson(my favourite). This gives you rapid feedback of any problems before they get under the radar and bite you from behind.

Ok, I'll stop now, I didn't intend to go for a "Longest answer" badge.