views:

424

answers:

7

We're using subversion, (the question could be applicable to many version control systems, but subversion is the one I really care about.)

Our repository layout looks like so:

(Layout A)

Web
  branches
  tags
  trunk
Libraries
  Foo
 branches
 tags
 trunk
  Bar
 branches
 tags
 trunk
WindowsClient
  branches
  tags
  trunk
DB
  branches
  tags
  trunk

The problem is that the unit of versioning is not equal to the unit of development -- I have to do multiple checkouts to get a buildable artifact, and when I branch, I have to branch multiple components (and check in in multiple places.)

This implies that we could instead move to a structure like this:

(Layout B)

Web
  branches
  tags
  trunk
    main
 libs
   Foo
   Bar
 DB
WindowsClient
  branches
  tags
  trunk
    main
 libs
   Foo
   Baz
 DB

But then we have duplicate copies of any shared libraries. We could map the shared libs in using svn:externals, but that's just an illusion -- They won't be branched when the containing project is.

A final option is this:

(Layout C)

branches
tags
trunk
  Web
  Libraries
    Foo
    Bar
  WindowsClient
  DB

This makes sure that libraries are branched along with their containing project, but at the cost that the unit of branching is the whole world. (This also implies that the unit of checkout is the whole world, which is annoying too.)

What I want is a repository layout (Layout D) that allows me to:

  • Branch a project and its dependent libraries all at once
  • Share libraries between projects

It would be nice if I could check out the project and its libraries in one checkout, but that is not nearly as important as the above.

So the question is:

Is there a Layout D, What is it, and how do I use it?

Edit: Since it appears there isn't a basic layout that will give me these properties, I'd be very interested in some kind of hook function to get me there. It would be especially nice if it will work with the TortoiseSVN (Windows GUI) client, since that's what we're using.

+1  A: 

There's no good answer for the question of "how to lay out my repository for this workflow?" because the software doesn't really support that. I'd suggest going with your Layout B, and branching the library code and switching the relevant svn:external to that branch as needed, or right away if your branches need to refer to an off-trunk version of the library.

I was going to suggest that Git handles this better, but it's not by much. Since its submodules refer to separate repositories slightly differently from externals, and each copy of the repository is a 'branch', that might be a slight improvement.

Novelocrat
+1  A: 

We could map the shared libs in using svn:externals, but that's just an illusion -- They won't be branched when the containing project is.

Actually, they will be branched if they are in the same repository and you used the relative external syntax, e.g. ^\mylib\trunk. Such external references are changed into normal (copied) folders. You have to explicitly pass --ignore-externals to svn copy to suppress this behavior, or else you'll end up with copies like in layout B. (edit: I was pretty sure it worked this way but I can't seem to reproduce that behavior. I must have been mistaken, sorry!)

The fact that externals don't always branch automagically doesn't have to be a problem. I would use layout B constructed with svn:externals (not copies), branch the project (with --ignore-externals), then after branching adapt the svn:externals to point to the correct library branches.

You can set up the externals to point to a specific revision (good for tight control; you decide when to upgrade to a new revision of the lbrary) or just track the HEAD (good for continuous integration, assuming that you have a build server set up).

Wim Coenen
:-( Your edit makes me sad. Oh well. Maybe I can do something similar with precommit hooks.
Sean McMillan
A: 

The way we have solved this is for shared external libraries used by several project, the shared library goes in its own repository with its own trunk/branches/tags.

We then have a build server build and publish integration builds, milestones and releases and the binary artefact is copied to a shared location and stored in version specific directories (these are backed up).

Part of the build script for a dependent project (normally run on demand as an init/update rather than as part of a standard build) then checks for new versions and grabs the binary. This has the advantage of consistent versioning of the shared artefact between dependent projects and reduces build time as all dependent projects can share the same version.

To help with the versioning we currently use Apache Ivy which supports things like transient dependencies (ie fetching the dependencies of a dependency) and version constraints (eg this project should only use Foo version 1.2.*).

KeeperOfTheSoul
Hmm... That would mean that I would have to explicitly version and release all of our internal libraries, then the build scripts would pull them back in. Most library changes would be adding a new function needed for a specific project -- our libraries really only exist to enable code sharing between or applications.
Sean McMillan
Right, but those are built by the CI server on every change and published as a snapshot. Alternatively, for local dev they are pushed to a local repository for that machine only, sort of like maven. This can be scripted by a master build file that builds each dependency in turn and pushes the artifacts to a local repository just for that build.
KeeperOfTheSoul
Unfortunately, we're a long way from Continuous Integration right now. :-(
Sean McMillan
+1  A: 

Go with option C, then do your checkouts like this:

svn co -N ...../branches/mybranch workingcopy
cd workingcopy
svn update Web Libraries

Now, when you do svn operations (including a plain "svn update"), it will just deal with the Web and Libraries directories.

Also read up on sparse directories.

retracile
You get the accepted answer because your answer will be hardest to screw up. While I don't enjoy the thought of teaching people to make sparse checkouts, It's the only (working) answer that doesn't translate to "Release and Version your libraries (with partial sugar)".
Sean McMillan
This is fine provided the libraries follow the same release cycle as the top-level product. But if the libraries are shared between multiple top-level products, you will need separately versioned subtress, as in Layout A.
gavinb
A: 

I'd suggest that it's your approach to these libraries which is causing your problems. You can change this if you start to think about libraries as separate projects in their own right. Think of them as having their own reasons for being, their own design and their own release cycles, just like the 3rd party libraries you may use for unit testing, xml readers, db access, etc.

Of course you will regularly have times where a feature in a project requires a new feature in a library. Implementing the library feature and making use of the library function are two independent tasks - they may be one business task but they're two development tasks. There's no need to tightly link the two activities together just because that's how the job came in. Checkout the library, change it, release it, then checkout your project and use the new release of the library in your project.

I feel strongly that having libraries separated out into their own trunks is a good thing - I can't stand it when I see multiple independently releasable projects under a single trunk. It smacks of poor design and having been backed into a corner by development cruft. But to separate them out you have to be able to release each project independently - to me, that's what having multiple projects means. But it's not a hard thing to do:

First a project uses externals to reference a specific released version of a library. That's the only way a project references a library. Doing this means that developers can make a new version of the library without breaking any of the projects using it, because all projects will be referencing the previous version. Projects get to control when they want to bring in new versions of libraries - the developers get to choose when they want to put in the effort of testing their code with the new version and when to take the pain of fixing any build issues the new library introduces.

When you explicitly change versions of a library like this, you also get an entry in your project that says "I'm now using this version of library X", which gives you a good sense of history in your project as to when things were working and exactly when things changed.

Now of course, this is all nice in theory, but in practice developers will sometimes have to reference unstable and unfinished versions of a library. That's fine - a developer can always switch their working copy to point to library trunk instead of a tag, or some development branch, and use the code from there (even work on it through there if they must, brrr). A switch is just a local edit, so will have no effect on the committed code. If the project development is on an unstable branch, then you can decide to make the switch more permanent by changing the externals reference until the branch is ready to be reintegrated, but this isn't something that would normally be done without an explicit reason.

And, finally, branching and tagging your project becomes a simple case of making a branch or tag of your main project - that's all. There's no need to worry about branching libraries - they take care of themselves. The process of making a change to a library doesn't change whether the project is in trunk, a development branch, or a maintenance release. And your libraries can themselves have development branches entirely independent of the main projects, as well as multiple supported versions, etc, down to whatever level of complexity you need and can support.

By using externals on your trunk or development branch, you can have a single checkout that builds your workspace in whatever structure you need. Because all libraries are under the main root, you can have multiple checkouts of multiple versions without getting clashes in the build.

I've found this system works pretty well, and after moving jobs to a place that doesn't work this way, I find myself pining for the previous way of working. There are issues, mainly to do with libraries depending on libraries and whether to have recursive externals or not. My take on that is to go recursive unless it causes a problem (or excessive pain), then move to a 'degenerate' model where the project has to know about certain 'deep' dependencies even if it doesn't use them directly. Also, decide where you're going to put your externals definitions and stick to it, nothing's more annoying than hunting for those svn:externals properties on random folders in different projects. Putting them on the root of trunk is fine.

Jim T
A: 

This is a hard problem that has no good cheep solution; the hi-end solution is to use a “software product lines” management system (e.g. pure::variants). However most of us don’t get to spend that match on a source code control system.

Therefore I would go with LayoutA – with each library versioned separately. However I would tend to put “trunk” under “branches” as it is a branch and I like to have all branch at the same distance from the top.

The next step rather depends on your build system, I am assuming Visual Studio here.

  • In the root of each product branch tree
  • Create a bat file that defines an environment variable that contains the name of the branch of each library you wish to use
  • Edit the Visual Studio project files to reference the libraries using these environment variables
  • Run the batch file from the Visual Studio command prompt before starting Visual Studio

You could also look at writing a custom MSBuild file rather then using a batch file. Or writing a tool that edits all the project files when you change version of a library.

If you only have 1 or 2 shared libraries and they are only change for one product at a time, .e.g. to add new methods for the project that is being worked on now. I would consider having a different branch of the library for each project, and use SVN 1.5 merge tacking to keep track of what is going on. (When changes are stable, merge to the truck, then merger from the truck to each projects branch when needed)

(If you have 100s of libraries, you have to track witch version of each library needs each other. This start to get very complex!)

I don’t like svn:external, as it is not clear from the file system on your local PC what is going on. However svn:external is a workable solution.

Ian Ringrose
A: 

Having been through a similar problem, I know your pain. It is a difficult problem to manage dependencies in a repository with hierarchical components.

Our project had several products (whatever you ship to customers) made up of various components (many of which are shared). We had each component with its own tags/branches/trunk trilogy, much like your Layout A (which is after all the recommended way).

We did use svn:externals to provide a way for each product to specify dependent components (and sub-components, etc), and at first it worked reasonably well. But eventually we ran into problems, such as what happens when you branch, what if one product needs to pin a depdency at a certain revision, how do you propagate tags through externals for configuration management (so you can actually reconstruct the same tree!), and so on. So svn:externals solve some problems but introduce others.

I ended up writing some scripts to manage this, but it was still a bit convoluted. Fortunately, you can use the Python-Subversion bindings to write Python apps to manipulate properties, so you can do things like have tags propagated through dependent components, and so on.

There is a project which is designed to address this very problem of dependent modules, called Piston. It looks like a very nice, generic tool for exactly this sort of problem. I did not deploy it in production, but at the time it looked like it would do most of what we need. And it certainly seems like a more flexible solution than externals provides (which is still a very manual process).

Bottom line: you could stick with Layout A, and use Piston to manage the dependencies so all the right versions of your libraries get assembled in your working directory.

gavinb
Piston doesn't seem to have any overview documentation. I only see "man pages" for the subcommands. Is there any documentation that describes how you are supposed to use it and what it does?
Sean McMillan
Yes, I see what you mean. These pages have a reasonable overview: http://www.rubyinside.com/advent2006/12-piston.html and http://www.yup.com/articles/2006/11/01/better-subversion-external-branch-management-with-piston
gavinb