Help with Estimating a Data Migration

Question

Our client has a completely custom CMS that was built in ASP 1.1 and later upgraded to 2.0. The database has over 200 tables, and unfortunately, there's no documentation for either the ASP code or the database. The original developers are unavailable for questioning, and no one at my company is familiar with the setup.

The majority of the database tables don't have timestamp columns, so it's difficult to determine from inspection alone which tables are in use and which aren't. Adding to the complexity of the task is that for each portal site in the CMS, custom functionality has been developed that uses individual database tables, and occasionally, stored procedures to work with data for an individual client site.

The original developer working on this project is gone, and it's been put on my plate unexpectedly. I'm tasked with moving all of the data in the existing CMS, including data for each module and for each client site, into a DotNetNuke installation with custom modules that we're developing. The estimate I've been handed is three weeks.

If anyone has attempted such a task before: is this doable in three weeks? I've never attempted such a massive data migration before, and any help with strategy would be helpful.

Answer 1

To start with the bad news: probably not. (I'm not thinking 3 calendar weeks: not ~140 manhours spread over a longer period).

The reason being that you have no documentation for anything that's a bit 'out of the ordinary' - strange behaviour from procedures, hacks to fix bugs, inconsistencies in the data, data pollution, etc. Given the fact there's no documentation, you cannot expect top-notch data validation or model either. (It may exist: it's just not guaranteed).

And that's just technical reasons: alarm bells are going off when you say 'custom modules that we're developing'. Your project is in way too much flux and will require too much talking between parties.

As to strategy: I'd identify the 80% common functionalitity that is at the core of each system and will require only 20% effort to convert. You have code and a db: put it on a test server and let a few users run through common actions. Put tracing/logging in your code and see what's hit. Put timestamp columns or an audit trail on your tables, and see what changes in your db.

Good luck!

Answer 2

I've been in a similar project a few years ago. Original schedule was three months, but it eventually took six months before we got into production. The system was a critical order processing application with a public website front end. About 40 back office users switched at once from the old system to the new one.

Most of the delays were due to the "custom modules" application development part, but migration took definitely more than 3 weeks. Source database had a decently normalized design with foreign key constraints and even some documentation. In the end, less than 50 tables were required in the migration. But it still took a while to reverse-engineer the old system. View definitions used in the database helped a lot in this processs.

Besides the extract phase, it also took considerable time to develop the transformation to the target. Transformations needed by the destination changed somewhat during the project. It certainly helps if your target schema is mostly fixed by now. Multiple corrections to data migration were also needed after the go-live, when there was already new data in the destination and it could not be completely reloaded anymore.

I would recommend that whatever technology you are using for the migration, develop it so that the process is automatic and easily repeatable. I used SSIS in my project. Until you go into production, you'll probably need to initialize and reload the test system's destination database multiple times. It is also helpful for application testing that there is some real data in a test system most of the time, even if only a few tables have been migrated.