SQL -> MongoDB Export Performance Issues

Question

I'm trying to set up an automated process to regularly transform and export a large MS SQL 2008 database to MongoDB.

There is not a 1-1 correspondence between tables in SQL and collections in MongoDB -- for example the Address table in SQL is translated into an array embedded in each customer's record in Mongo and so on.

Right now I have a 3 step process:

1. Export all the relevant portions of the database to XML using a FOR XML query.
2. Translate XML to mongoimport-friendly JSON using XSLT
3. Import to mongo using mongoimport

The bottleneck right now seems to be #2. XML->JSON conversion for 3 million customer records (each with demographic info and embedded address and order arrays) takes hours with libxslt.

It seems hard to believe that there's not already some pre-built way to do this, but I can't seem to find one anywhere.

Questions:

A) Are there any pre-existing utilities I could use to do this?
B) If no, is there a way I could speed up my process?
C) Am I approaching the whole problem the wrong way?

Answer 1

Another approach is to go through each table and add information to mongo on a record by record basis and let Mongo do the denormalizing! For instance to add each phone number, just go through the phone number table and do a '$addToSet' for each phone number to the record.

You can also do this in parallel and do tables separately. This may speed things up but may 'fragment' the mongo database more.

You may want to add any required indexes before you start, otherwise adding the indexes at the end may be a large delay.