views:

4784

answers:

12

First, I know about this: http://stackoverflow.com/questions/51217/how-would-you-organize-a-subversion-repository-for-in-house-software-projects Next, the actual question: My team is restructuring our repository and I'm looking for hints on how to organize it. (SVN in this case). Here's what we came up with. We have one repository, multiple projects and multiple svn:externals cross-references

\commonTools /*tools used in all projects. Referenced in each project with svn:externals*/
   \NUnit.v2.4.8
   \NCover.v.1.5.8
   \<other similar tools>
\commonFiles /*settings strong name keys etc.*/
   \ReSharper.settings
   \VisualStudio.settings
\thrash /*each member of the team has thrash for samples, experiments etc*/
   \user1
   \user2
\projects
   \Solution1 /*Single actual project (Visual Studio Solution)*/
      \trunk
         \src
             \Project1 /*Each sub-project resulting in single .dll or .exe*/
             \Project2
         \lib
         \tools
         \tests
         \Solution1.sln
      \tags
      \branches
   \Solution2
      \trunk
         \src
             \Project3 /*Each sub-project resulting in single .dll or .exe*/
             \Project1 /*Project1 from Solution1 references with svn:externals*/
         \lib
         \tools
         \tests
         \Solution2.sln
      \tags
      \branches

To clear the vocabulary: Solution means single product, Project is a Visual Studio Project (that results in a single .dll or single .exe)

That's how we plan to lay out the repository. The main issue is, that we have multiple Solutions, but we want to share Projects among Solutions. We thought that there is no point really in moving those shared Projects to their own Solutions, and instead we decided to use svn:externals to share Projects among Solutions. We also want to keep common set of tools and 3rd party libraries in one place in the repository, and them reference them in each Solution with svn:externals.

What do you think about this layout? Especially about the use of svn:externals. It's not an ideal solution, but considering all pros and cons, it's the best we could think of. How would YOU do it?

+2  A: 

I believe that Pragmatic Version Control using Subversion has everything you need to organize your repository.

Fabio Gomes
Now in a 2nd edition: http://is.gd/Eo4
balexandre
A: 

I have a question - not an answer

In your scheme how do you refer to files from a different project in a relative way - given that there is the trunk/tags/branches thing?

Tim
You should not be having a conversation via answers. That is what these comments are for.
raven
So sue me. It was relevant to the whole thread.
Tim
Don't ever reference one project from another, unless you like/want a maintenance hell. Please see my detailed answer.
Rob Williams
@Rob "Ever" is such a strong word. It works pretty damn well with medium-sized C# projects that are co-developed.
romkyns
+3  A: 

We've set ours up to almost exactly match what you've posted. We use the general form:

\Project1
   \Development (for active dev - what you've called "Trunk", containing everything about a project)
   \Branches (For older, still-evolving supported branches of the code)
       \Version1
       \Version1.1
       \Version2
   \Documentation (For any accompanying documents that aren't version-specific

While I suppose not as complete as your example, it's worked well for us and lets us keep things separate. I like the idea of each user having a "Thrash" folder as well - currently, those types of projects don't end up in Source control, and I've always felt that they should.

rwmnau
I'm surprised that you have a separate directory for documents that don't change between versions...I've never had the pleasure of working on such a product! : )
ARKBAN
A: 

@Tim, As I wrote, this is a proposition. We didn't yet implement that idea.

This is a good question. We didn't think about that. We just assumed that we will only be referring to a known revision. The referenced Projects are read/write in the Solution they originally belong to. In all other projects they are to be read only. So I guess this should not be an issue, and when there are bugfixes introduced to the referenced Project we will just up the revision number in the referencing projects, I guess.

That is a great question though, and one we didn't think much about yet. Do you have a good solution for that?

Krzysztof Koźmic
You should not be having a conversation via answers. That is what these comments are for.
raven
Please see my answer for details.
Rob Williams
A: 

I have a similar layout but my trunk, branches, tags all the way at the top. So: /trunk/main, /trunk/utils, /branches/release/ etc etc. This ended up being really handy when we wanted to try out other version control systems because many of the translation tools worked best with the basic textbook svn layout.

A: 

I think the main disadvantage of the proposed structure is that the shared projects will only be versioned with the first solution they were added to (unless svn:externals is fancier than I am imagining). For example, when you create a branch for the first release of Solution2, Project1 won't be branched since it lives in Solution1. If you need to build from that branch at a later time (QFE release), it will use the latest version of Project1 rather than version of Project1 at the time of the branch.

For this reason, it may be advantageous to put the shared projects in one or more shared solutions (and thus top-level directories in your structure) and then branch them with each release of any solution.

C. Dragon 76
You are right to some extent. But we can update the reference if we want to. And puting shared Projects into their own Solution does not make much sense either. Although I would love to find a better solution than svn:externals all over the place.
Krzysztof Koźmic
What do you mean by "update the reference if we want to"? I don't see how you would be able to branch Project1 (which seems desirable whenever you branch Solution2) without branching Solution1.
C. Dragon 76
Please see my detailed answer, particularly to NOT put Visual Studio solutions into source control.
Rob Williams
A: 

Why have it all in one repository? Why not just have a separate repository for each project (I mean "Solution")?

Well, at least I've used to the one-project-per-repository-approach. Your repository structure seems overcomplicated to me.

And how many project do you plan to put into this one big repository? 2? 3? 10? 100?

And what do you do when you cancel the development of one project? Just delete it from repository tree so that it will become hard to find in the future. Or leave it lying around forever? Or when you want to move one project to another server altogether?

And what about the mess of all those version numbers? The version numbers of one project going like 2, 10, 11, while the other goes like 1, 3, 4, 5, 6, 7, 8, 9, 12...

Maybe I'm foolish, but I like one project per repository.

Rene Saarsoo
1. One repository is a company policy, can't change that.2. We will have around dozen Solutions.3. by version numbers you mean revisions? That is not an issue to us.
Krzysztof Koźmic
A good project structure should be oblivious to the rest of the repository structure, particularly with regard to one or many repositories. Please see my detailed answer.
Rob Williams
Please note that having multiple repositories in many (most?) source control tools can be VERY expensive, such as when you implement security.
Rob Williams
A: 

To add to the relative path issue:

I am not sure it is a problem:
Just checkout Solution1/trunk under the directory named "Solution1", ditto for Solution2: the goal of 'directories' actually representing branches is to not be visible once imported into a workspace. Hence relative paths are possible between 'Solution1' (actually 'Solution1/trunk') and 'Solution2' (Solution2/trunk).

VonC
This would break very easily, please see my detailed answer.
Rob Williams
A: 

RE: the relative path and shared file issue -

It seems that this is svn specific, but that is not a problem. One other person already mentioned separate repositories and that is probably the best solution I can think of in the case where you have different projects referring to arbitrary other projects. In the case where you have no shared files then the OP solution (As well as many others) will work fine.

We're still working this out and I have 3 different efforts (different clients) that I have to resolve right now since I took over setting up either non-existent or poor version control.

Tim
Having projects reference other projects creates a maintenance nightmare because the dependencies grow exponentially and the references are VERY fragile. Please see my detailed answer.
Rob Williams
A: 

@Tim and @Rene I commented on Rene' post, but it's not really visible. One repository/team is a company policy. Can't change that. I also don't see how that would improve the maintainability of it all, which is my main concern. We decided to use svn:externals because the only alternative was to have binary references to compiled versions of the other projects. This is not a good solution considering even the simplest scenario: A patch is introduced to the referenced Project, that fixes a bug. Now I have to compile the project get the binary and copy it manually to lib folder of all projects that reference it. with svn:externals I just apply the patch, commit the changes to the main Solution of that Project, and them I update the svn:externals reference (if I'm bound to specific revison) to the fixed version. This also gives me the advantage of having the code, so I can step in with the debugger, and I can do debug/release versions etc.

Krzysztof Koźmic
I provide an alternative, and I recommend avoiding Subversion externals like the plague. Please see my detailed answer.
Rob Williams
+34  A: 

If you follow my recommendations below (I have for years), you will be able to:

-- put each project anywhere in source control, as long as you preserve the structure from the project root directory on down

-- build each project anywhere on any machine, with minimum risk and minimum preparation

-- build each project completely stand-alone, as long as you have access to its binary dependencies (local "library" and "output" directories)

-- build and work with any combination of projects, since they are independent

-- build and work with multiple copies/versions of a single project, since they are independent

-- avoid cluttering your source control repository with generated files or libraries

I recommend (here's the beef):

  1. Define each project to produce a single primary deliverable, such as an .DLL, .EXE, or .JAR (default with Visual Studio).

  2. Structure each project as a directory tree with a single root.

  3. Create an automated build script for each project in its root directory that will build it from scratch, with NO dependencies on an IDE (but don't prevent it from being built in the IDE, if feasible).

  4. Consider nAnt for .NET projects on Windows, or something similar based on your OS, target platform, etc.

  5. Make every project build script reference its external (3rd-party) dependencies from a single local shared "library" directory, with every such binary FULLY identified by version: %DirLibraryRoot%\ComponentA-1.2.3.4.dll, %DirLibraryRoot%\ComponentB-5.6.7.8.dll.

  6. Make every project build script publish the primary deliverable to a single local shared "output" directory: %DirOutputRoot%\ProjectA-9.10.11.12.dll, %DirOutputRoot%\ProjectB-13.14.15.16.exe.

  7. Make every project build script reference its dependencies via configurable and fully-versioned absolute paths (see above) in the "library" and "output" directories, AND NO WHERE ELSE.

  8. NEVER let a project directly reference another project or any of its contents--only allow references to the primary deliverables in the "output" directory (see above).

  9. Make every project build script reference its required build tools by a configurable and fully-versioned absolute path: %DirToolRoot%\ToolA\1.2.3.4, %DirToolRoot%\ToolB\5.6.7.8.

  10. Make every project build script reference source content by an absolute path relative to the project root directory: ${project.base.dir}/src, ${project.base.dir}/tst (syntax varies by build tool).

  11. ALWAYS require a project build script to reference EVERY file or directory via an absolute, configurable path (rooted at a directory specified by a configurable variable): ${project.base.dir}/some/dirs or ${env.Variable}/other/dir.

  12. NEVER allow a project build script to reference ANYTHING with a relative path like .\some\dirs\here or ..\some\more\dirs, ALWAYS use absolute paths.

  13. NEVER allow a project build script to reference ANYTHING using an absolute path that does not have a configurable root directory, like C:\some\dirs\here or \\server\share\more\stuff\there.

  14. For each configurable root directory referenced by a project build script, define an environment variable that will be used for those references.

  15. Attempt to minimize the number of environment variables you must create to configure each machine.

  16. On each machine, create a shell script that defines the necessary environment variables, which is specific to THAT machine (and possibly specific to that user, if relevant).

  17. Do NOT put the machine-specific configuration shell script into source control; instead, for each project, commit a copy of the script in the project root directory as a template.

  18. REQUIRE each project build script to check each of its environment variables, and abort with a meaningful message if they are not defined.

  19. REQUIRE each project build script to check each of its dependent build tool executables, external library files, and dependent project deliverable files, and abort with a meaningful message if those files do not exist.

  20. RESIST the temptation to commit ANY generated files into source control--no project deliverables, no generated source, no generated docs, etc.

  21. If you use an IDE, generate whatever project control files you can, and don't commit them to source control (this includes Visual Studio project files).

  22. Establish a server with an official copy of all external libraries and tools, to be copied/installed on developer workstations and build machines. Back it up, along with your source control repository.

  23. Establish a continuous integration server (build machine) with NO development tools whatsoever.

  24. Consider a tool for managing your external libraries and deliverables, such as Ivy (used with Ant).

  25. Do NOT use Maven--it will initially make you happy, and eventually make you cry.

Note that none of this is specific to Subversion, and most of it is generic to projects targeted to any OS, hardware, platform, language, etc. I did use a bit of OS- and tool-specific syntax, but only for illustration--I trust that you will translate to your OS or tool of choice.

Additional note regarding Visual Studio solutions: don't put them in source control! With this approach, you don't need them at all or you can generate them (just like the Visual Studio project files). However, I find it best to leave the solution files to individual developers to create/use as they see fit (but not checked in to source control). I keep a Rob.sln file on my workstation from which I reference my current project(s). Since my projects all stand-alone, I can add/remove projects at will (that means no project-based dependency references).

Please don't use Subversion externals (or similar in other tools), they are an anti-pattern and, therefore, unnecessary.

When you implement continuous integration, or even when you just want to automate the release process, create a script for it. Make a single shell script that: takes parameters of the project name (as listed in the repository) and tag name, creates a temporary directory within a configurable root directory, checks out the source for the given project name and tag name (by constructing the appropriate URL in the case of Subversion) to that temporary directory, performs a clean build that runs tests and packages the deliverable. This shell script should work on any project and should be checked into source control as part of your "build tools" project. Your continuous integration server can use this script as its foundation for building projects, or it might even provide it (but you still might want your own).

@VonC: You do NOT want to work at all times with "ant.jar" rather than "ant-a.b.c.d.jar" after you get burned when your build script breaks because you unknowingly ran it with an incompatible version of Ant. This is particularly common between Ant 1.6.5 and 1.7.0. Generalizing, you ALWAYS want to know what specific version of EVERY component is being used, including your platform (Java A.B.C.D) and your build tool (Ant E.F.G.H). Otherwise, you will eventually encounter a bug and your first BIG problem will be tracking down what versions of your various components are involved. It is simply better to solve that problem up front.

Rob Williams
So many points to criticize... suffice to say this is *not* an universal recipe! Point 5 and 6 in particular are oh so wrong when the project is big and the number of third parties is important: you want to work at all times with 'ant.jar', not 'ant1.5.4.jar', or product myProduct.exe, not 1.3.exe
VonC
Still, +1 for many other points you are making which are valid and speaks highly for your vast experience on the topic.
VonC
I would love to hear and interact with your criticisms--each and every point is based on solving bad experiences with large projects. For example, addressing the issue of which versions are represented by the Xxx.jar and Yyy.exe, especially when there are literally a dozen copies being referenced.
Rob Williams
@VonC: I suspect that your desire to reference myProduct.exe might be for developing against the latest "snapshot", such as is represented by Maven. In that case, I would tend to agree, and using a convention similar to Maven may be most appropriate.
Rob Williams
@Rob - Can you elaborate on your 'externals antipattern' theme? I've raised it as a question here: http://stackoverflow.com/questions/338824/are-subversion-externals-an-antipattern
Ken
#12 is so against any common sense. That means everyone has to have the same file system and locate the stuff on the same drive. Many vcs's allow you to create mapped drives to the data and this would screw it over.And I just wonder what would have happened to my system recently, when a certain shared netword resource was supposed to be mapped on M: but for some reason my machine didn't let me map to that, so I had to use N: instead...
Makis
@Makis: You would be correct, IF #12 were not balanced by #13. Every reference to a file or directory within each project should be made via an absolute path that starts with a configurable root directory variable, e.g. ${basedir}/sub/dir/file.txt in Ant.
Rob Williams
Why do you prefer Ant to Maven2? My experience is quite opposite.
Petr Macek
There is good content here and people should be aware of all the options they have. However, I don't agree that this is a one-size-fits-all solution.
Tim
+1  A: 

I am trying to implement a similar scheme but I found that there is a very annoying limitation which, it seems, would make it impossible to achieve what we need: svn-externals does not allow absolute paths, nor relative path with ".." in the target directory. If you want to have a common directory at the same level or above your local Solutions directory, you have a problem... It looks like the local (target) commonTool must always be somewhere under Solution. The newer SVN 1.5 syntax, although it addresses a similar problem when specifying the source URL, does nothing different for the specification of the target location.

So what bugs me is the requirement to have common modules grafted under every project instead of at the same level or a higher level.

Using svn:externals forces to have structure alike this one:

  • Project1

    • src
    • docum
    • commonTools
      • networking_1.2.4
      • UIControls_1.1.3
  • Project2

    • src
    • docum
    • commonTools
      • networking_1.2.4
      • UIControls_1.1.5

Instead of a more "shared" and space-saving layout:

  • commonTools

    • networking_1.2.4
    • UIControls_1.1.3
    • UIControls_1.1.5
  • Project1

    • src
    • docum
    • ... and project references to /commonTools/networking_1.2.4 and /commonTools/UIControls_1.1.3 (referring to absolute or relative paths)
  • Project2

    • src
    • docum
    • ... and project references to /common/networking_1.2.4 and /common/UIControls_1.1.5 (referring to absolute or relative paths)

*I understand though that such an convention (with version-qualified directory names) is more suited to third-party libraries; I would not use explicit version numbers in dir names like that for our own code base AND I would put those modules under each project using them.)

So how are you dealing with externals and 3rd party libraries?

SClark
I think you're abusing the structure here. Have you tried restructuring your layout? Or plase elaborate a little bit your specific scenario
Krzysztof Koźmic
I have clarified the case and provided example. Thanks.
SClark