Would it be a good idea to start converting forms into .NET one at a time which you would then invoke from the VB6 app via COM-interop.
This way, by the end of the process you would just convert the 'shell' of the VB6 application into a new .NET app, and all your forms are ready to go in .NET.
Is there a better strategy?
If it were me, I'd just rip the Band-Aid right off. I don’t see any benefit in having an in-between state [.NET Forms and COM-interop] for your application because it just adds unneeded complexity.
We have a VB6 application which is being ported to .NET either, and we use the COM-Interop strategy. All new features can be implemented in .NET, only the GUI-stuff remains VB; at the same time, we can develop a new GUI independently.
If you don't already know this, you can do COM Interop without using the registry (as this caused some problems for us) with Registration-Free COM Interop:
The piecemeal approach you describe will create extra work because you will be trying to get VB6 and .Net code to co-exist, which is fraught with problems on anything but the simplest applications. Somewhere down the road you are likely to trip over a gotcha which may be a showstopper.
I would recommend the following approach (based on successfully migrating a 600,000 line VB6 app to .Net):
Make sure your existing VB6 code base is properly version controlled and labelled. Write regression tests for your VB6 code base, preferably automated. Take a known VB6 code label baseline and migrate it as a single entity to .Net. Your customers continue to use the VB6 version. Run your regression tests on the migrated code. When all the tests pass, apply to the .Net code any VB6 changes that occurred since you took the original VB6 baseline. Deliver to UAT and then live.
There's lots of advice on strategies for conversion.
Would you do a large complex data conversion one-table at a time, with your systems using one, the other, or both data models concurently for an extended period? Clearly this is asking for trouble and complexity. Best practice for large data conversions is to do them in a way that brings the entire data model to its desired end state within the shortest possible outage. The same reasoning applies to large-scale code conversions. Doing them piecemeal over and extended period of time is asking for trouble in terms of increases labor costs and technical risk.
IMO, a better approach is to formulate the end-state development and architecture standards for your .NET code and then invest in a process that helps you efficiently rewrite your system in a way that conforms to those standards and accurately preserves legacy business rules and functional behavior. Long transitions and complex hybrid/intermediate solutions are only a stop-gap at best cause business problems and project failure at worst -- they should be avoided. A better approach will allow you to deliver the legacy software to the new platform in internally consistent, independent, and well-formed pieces. Furthermore, delivering the migration in fewer, larger pieces will be more efficient and less disruptive than many little pieces.
The key to making this approach viable is to use next generation VB6/COM/ASP to .NET tools that allow you to iteratively calibrate, customize, and verify an automated rewrite process that balances automated conversion with manual work. The tools from Great Migrations are specifically designed to enable this methodology. We call it the "tool-assisted rewrite". We have used this approach on several large migrations projects including upgrading an application portfolio of 1.2M LOC of VB6/COM to re-engineered C#/.NET.
Disclaimer: I work for Great Migrations.