Why would a long Rake task just stop, then start again?

Question

I have a complex legacy data migration problem. MS Access data going into MySQL. I'm using a Rake task. There's a lot of data and it requires a lot of transforming and examining. The Rake task is hundreds of lines across about 12 files. The whole thing takes about two hours to run. It has to run on Windows (I'm using XP VMware VM hosted on an OS X Leopard system) because the Ruby libraries that can talk to MS Access only work on Windows.

I'm finding that sometimes, not every time, I'll start the task and come back later and it will be stalled. No error message. I put numerous print statements in it so you should see lots of reporting going by, but there's just the last thing it did just sitting there. "11% done" or whatever.

I hit Ctrl-C to and instead of going back to the command prompt, the task starts up again where it left off, reported output starts going by again.

I'm sorry for the abstract question, but I'm hoping someone might have an idea or notion of what's happening. Maybe suggestions for troubleshooting.

Answer 1

Well, if the access side seems to be freezing, consider shoving the data into MySql, and see if that eliminates this problem. In other words, the data has to go over eventually, you might as move the data into that system from the get go. There is a number of utilities around that allow you to move the data into MySql (or just export the access data to CSV files).

So, you not doing data transformations during that transfer of data while you move it into MySql (so, it not a labor nor programming cost of time here…just transfer the data).

Once you have the data in MySql, then your code is doing the migration (transformation) of data from one MySql database (or table) to another. And, you out of a VM environment and running things in a native environment. (faster performance, likely more stable).

So, I would vote to get your data over into MySql..then you down to a single platform.

The less systems involved, the less chance of problems.