git submodule svn external

Question

Let's say I have 3 git repositories, each with a lib and tests folder in the root. All 3 repositories are part of what I want to be a single package, however it is important to me to keep the repositories separate.

I am new to git coming from svn, so I have been reading up on submodules and how they differ from svn:externals. In SVN I could have a single

lib/vendor/package

directory, and inside package I could setup 3 externals pointing to each of my 3 repositories lib directory, renaming it appropriately like

lib/vendor/package/a  -> repo1/lib
lib/vendor/package/b  -> repo2/lib
lib/vendor/package/c  -> repo3/lib

but from my understanding this is not possible with git. Am I missing something?

Really I'm hoping this can be solved in one of two ways.

1. Someone will point out how to create a 4th git repository which has the other 3 as submodules organized as I have mentioned above (where I can have an a, b, and c folder inside the root)
2. Someone will point out how to set this up using svn:externals in combination with githubs svn support, referencing the lib directory within each git repository (from my understanding this is impossible)

Answer 1

You can have a forth 'lib' main Git repo, with:

• lib/vendor/package content
• 3 submodules (see "true nature of submodules")

But this reference is totally independent from any 'svn:external' property you may have setup in a mirror SVN repo.

So if you have 3 SVN repos already, you can git-svn them, publish them on GitHub, and then create a fourth repo on GitHub in which you will add those 3 Git repos as submodules, following the submodules tutorial (supposing here you have all 4 repos already on GitHub)

$ mkdir -p lib/vendor/package
$ cd lib/vendor/package
$ for package in a b c d; do
$ git submodule add git://github.com/path/to/$package.git $package
$ done
$ cd ..
$ git commit -m "lib with submodules"
$ git push

Answer 2

Hi VonC-

Thanks for the reply. I had actually tried to follow the submodules tutorial you linked to, but I run into the following problem.

Doing things as shown above, instead of a mapping like

lib/vendor/package/a  -> repo1/lib
lib/vendor/package/b  -> repo2/lib
lib/vendor/package/c  -> repo3/lib

I am left with

lib/vendor/package/a  -> repo1
lib/vendor/package/b  -> repo2
lib/vendor/package/c  -> repo3

this is not ideal since now to access ClassA inside repo1's lib folder, the path is

lib/vendor/package/a/lib/ClassA

when I'm really trying to get (and this is possible with svn:externals)

lib/vendor/package/a/ClassA

since a above is actually repo1/lib, and not the root directory of repo1.

Something like this is important since, with PHP5.3 for example, using the SplClassLoader ( http://gist.github.com/221634 ), it requires a namespace-to-directory mapping like

\Package\a\ClassA  -> lib/vendor/package/a/ClassA

this is where my conceptual misunderstanding is, how to setup that 4th git repository to allow my directory mappings like above.

Answer 3

Any other thoughts VonC? Anyone?

Answer 4

You are right, Git submodules can not directly do exactly what you want. It works in SVN because the root of a repository, branches, and any subdirectory thereof are the same kind of object. In Git, a repository, a branch, and a directory are all distinct kinds of objects (you can not use a directory as a full repository or as a branch).

There are a couple of indirect ways to accomplish what you want though.

Using Submodules and Symlinks

The core of a Git submodule is a clone of another repository in the work tree of the “superproject”^*. Git only clones full repositories. It is not possible to clone just a single subdirectory out of an existing repository^†.

^* Normal submodules also require a special reference in the superproject's commits/index and (normally) an entry in the superproject's .gitmodules file. It is possible to have non-tracked clones of other repositories in an unrelated working tree, but such usage does not create a submodule.
^† Git 1.7.0 and later has a “sparse checkout” feature, but it would not help to relocate the lib directory the top level of each submodule clone.

You might, however be able to use Git's support for symbolic links to do something that is fairly close:

#
# Make the lib directory of each submodule appear in the superproject as
# lib/vendor/packages/$submod_name
#
# With this structure in each of the submodules (a, b, c):
#
#    lib/
#    tests/
#
# We end up with this structure in the superproject:
#
#    lib/
#        vendor/
#            packages/
#                a     (a symlink to ../../../_submodules/a/lib)
#                b     (a symlink to ../../../_submodules/b/lib)
#                c     (a symlink to ../../../_submodules/c/lib)
#    _submodules
#        a/            (a Git submodule)
#            lib/
#            tests/
#        b/            (a Git submodule)
#            lib/
#            tests/
#        c/            (a Git submodule)
#            lib/
#            tests/
#
add_one() {
    dir=lib/vendor/package
    dest="$dir/$1"
    # use fewer ".."s to put the _submodules closer to the symlinks
    s=../../../_submodules/"$1"
    git submodule add "$2" "$dir/$s"
    ln -s "$s"/lib "$dest"
    git add "$dest"
}

cd "$main_repo_toplevel"
mkdir -p lib/vendor/package
add_one a [email protected]:user/package-a.git
add_one b git://public.example.com/work/package-b-dev.git
add_one c ssh://special.example.com/foo.git

Using git subtree

apenwarr's git subtree can split off and merge parts of repositories (i.e. individual subdirectories; it is a wrapper around “subtree merging” with other nice features). The first step would be to extract the history of lib in each of your sub-projects. Then, either directly use the extracted history as a submodule, or use git subtree to do a subtree merge into your main repository. Either way, this would introduce an extra step (re-extracting the lib history) before you could integrate changes from a sub-project into your main repository.