What is the correct way to display UI during a custom installer action?
I would like my UI to be modal on the install dialog, or alternatively, I'd like a way to display text/progress from my custom action in the installer dislog.
The installer is a VS2005 setup project and the custom action is a C# Installer-derived class.
Displaying any kind of non-standard UI would require changes to the UI handler object. This isn't trivial, and the implementation depends on the toolkit you use to author your MSIs: I'm not sure it's even possible with VS setup projects.
Displaying simple status/progress messages and logging to the MSI log isn't too hard to do from a custom action, though, at least not using the Windows Installer XML (WiX) toolset, which is what I use myself for this purpose.
When authoring your custom actions with WiX, you get access to the active installer session through the Microsoft.Deployment.WindowsInstaller.Session object, which has 'Log' (writes a message to the log, if logging is enabled) and 'Message' (performs any enabled logging operations and defers execution to the UI handler object associated with the engine) functions, amongst many other goodies.
If you're currently already creating your custom actions in C#, you may be able to find something similar in your current environment (I've never worked with VS.net installer projects, so I'm not exactly sure how they work -- I'm quite surprised actually that these allow you to create managed custom actions...). Otherwise, I'd definitely recommend looking into WiX for custom actions: these work with any MSI authoring environment, and are quite flexible.