Split large repo into multiple subrepos and preserve history (Mercurial)

Question

We have a large base of code that contains several shared projects, solution files, etc in one directory in SVN. We're migrating to Mercurial. I would like to take this opportunity to reorganize our code into several repositories to make cloning for branching have less overhead. I've already successfully converted our repo from SVN to Mercurial while preserving history. My question: how do I break all the different projects into separate repositories while preserving their history?

Here is an example of what our single repository (OurPlatform) currently looks like:

/OurPlatform
---- Core
---- Core.Tests
---- Database
---- Database.Tests
---- CMS
---- CMS.Tests
---- Product1.Domain
---- Product1.Stresstester
---- Product1.Web
---- Product1.Web.Tests
---- Product2.Domain
---- Product2.Stresstester
---- Product2.Web
---- Product2.Web.Tests
==== Product1.sln
==== Product2.sln

All of those are folders containing VS Projects except for the solution files. Product1.sln and Product2.sln both reference all of the other projects. Ideally, I'd like to take each of those folders, and turn them into separate Hg repos, and also add new repos for each project (they would act as parent repos). Then, If someone was going to work on Product1, they would clone the Product1 repo, which contained Product1.sln and subrepo references to ReferenceAssemblies, Core, Core.Tests, Database, Database.Tests, CMS, and CMS.Tests.

So, it's easy to do this by just hg init'ing in the project directories. But can it be done while preserving history? Or is there a better way to arrange this?

EDIT::::

Thanks to Ry4an's answer, I was able to accomplish my goal. I wanted to share how I did it here for others.

Since we had a lot of separate projects, I wrote a small bash script to automate creating the filemaps and to create the final bat script to actually do the conversion. What wasn't completely apparent from the answer, is that the convert command needs to be run once for each filemap, to produce a separate repository for each project. This script would be placed in the directory above a svn working copy that you have previously converted. I used the working copy since it's file structure best matched what I wanted the final new hg repos to be.

#!/bin/bash

# this requires you to be in: /path/to/svn/working/copy/, and issue: ../filemaplister.sh ./

for filename in *
do
  extension=${filename##*.} #$filename|awk -F . '{print $NF}'
  if [ "$extension" == "sln" -o "$extension" == "suo" -o "$extension" == "vsmdi" ]; then
    base=${filename%.*}
    echo "#$base.filemap" >> "$base.filemap"
    echo "include $filename" >> "$base.filemap"
    echo "C:\Applications\TortoiseHgPortable\hg.exe convert --filemap $base.filemap ../hg-datesort-converted ../hg-separated/$base > $base.convert.output.txt" >> "MASTERGO.convert.bat"
  else
    echo "#$filename.filemap" >> "$filename.filemap"
    echo "include $filename" >> "$filename.filemap"
    echo "rename $filename ." >> "$filename.filemap"
    echo "C:\Applications\TortoiseHgPortable\hg.exe convert --filemap $filename.filemap ../hg-datesort-converted ../hg-separated/$filename > $filename.convert.output.txt" >> "MASTERGO.convert.bat"  
  fi  
done;

mv *.filemap ../hg-conversion-filemaps/
mv *.convert.bat ../hg-conversion-filemaps/

This script looks at every file in an svn working copy, and depending on the type either creates a new filemap file or appends to an existing one. The if is really just to catch misc visual studio files, and place them into a separate repo. This is meant to be run on bash (cygwin in my case), but running the actual convert command is accomplished through the version of hg shipped with TortoiseHg due to forking/process issues on Windows (gah, I know...).

So you run the MASTERGO.convert.bat file, which looks at your converted hg repo, and creates separate repos using the supplied filemap. After it is complete, there is a folder called hg-separated that contains a folder/repo for each project, as well as a folder/repo for each solution. You then have to manually clone all the projects into a solution repo, and add the clones to the .hgsub file. After committing, an .hgsubstate file is created and you're set to go!

With the example given above, my .hgsub file looks like this for "Product1":

Product1.Domain = /absolute/path/to/Product1.Domain
Product1.Stresstester = /absolute/path/to/Product1.Stresstester
Product1.Web = /absolute/path/to/Product1.Web
Product1.Web.Tests = /absolute/path/to/Product1.Web.Tests

Once I transfer these repos to a central server, I'll be manually changing the paths to be urls.

Also, there is no analog to the initial OurPlatform svn repo, since everything is separated now.

Thanks again!

Answer 1

This can absolutely be done. You'll want to use the hg convert command. Here's the process I'd use:

1. convert everything to a single hg repository using hg convert with a source type of svn and a dest type of hg (it sounds like you've already done this step)
2. create a collection of filemap files for use with hg convert's --filemap option
3. run hg convert with source type hg and dest type hg and the source being the mercurial repo created in step one -- and do it for each of the filemaps you created in step two.

The filemap syntax is shown in the hg help convert output, but here's the gist:

The filemap is a file that allows filtering and remapping of files and
directories. Comment lines start with '#'. Each line can contain one of
the following directives:

  include path/to/file

  exclude path/to/file

  rename from/file to/file

So in your example your filemaps would look like this:

# this is Core.filemap
include Core
rename Core .

Note that if you have an include that the exclusion of everything else is implied. Also that rename line ends in a dot and moves everything up one level.

# this is Core.Tests
include Core.Tests
rename Core.Tests .

and so on.

Once you've created the broken-out repositories for each of the new repos, you can delete the has-everything initial repo created in step one and start setting up your subrepo configuration in .hgsub files.