In VS 2008 setup/deployment project, is there any way to add a custom error dialog (or customize the existing one) to handle errors encountered during the installation process?
You can add additional dialogs, but I dont think there is a way you can redirect errors to them. With the Studio setup projects, I think you are stuck with the built in error handling.
You would need to use a 3rd party product like InstallShield, or hand edit the installer (MSI) to get the desired functionality.
If I understand you correctly:
Not exactly for your software but, you can handle errors for your software's prerequisites which cause errors most of the time. You must first go to the location of prerequisite package file. Normally, that would be "C:\Program Files\Microsoft SDKs\Windows\v6.0A\Bootstrapper\Packages\". Find your package folder, move in, there is a folder named "en" and inside it there is an xml file named "package.xml". Open it with a text editor and you will see:
<InstallChecks>
<RegistryCheck Property="DotNet35SP" Key="HKLM\Software\Microsoft\NET Framework Setup\NDP\v3.5" Value="SP" />
</InstallChecks>
<!-- Defines how to invoke the setup for .NET Framework redist -->
<Commands Reboot="Defer">
<Command PackageFile="dotNetFx35setup.exe"
Arguments=' /lang:enu /passive /norestart'
EstimatedInstalledBytes="30000000"
EstimatedTempBytes="30000000">
<!-- These checks determine whether the package is to be installed -->
<InstallConditions>
<!-- This indicates .NET Framework is already installed -->
<BypassIf Property="DotNet35SP" Compare="ValueGreaterThanOrEqualTo" Value="1"/>
<!-- Block install if user does not have admin privileges -->
<FailIf Property="AdminUser" Compare="ValueEqualTo" Value="false" String="AdminRequired"/>
<!-- Block install on less than Windows XP SP2 -->
<FailIf Property="VersionNT" Compare="VersionLessThan" Value="5.1.2" String="InvalidPlatformWinNT"/>
<!-- Block install on W2K3 with no service pack -->
<FailIf Property="VersionNT" Compare="VersionEqualTo" Value="5.2.0" String="InvalidPlatformWinNT"/>
<!-- Block install if the platform is IA-64 -->
<FailIf Property="ProcessorArchitecture" Compare="ValueEqualTo" Value="IA64" String="InvalidPlatformArchitecture" />
</InstallConditions>
<ExitCodes>
<ExitCode Value="0" Result="Success"/>
<ExitCode Value="1602" Result="Fail" String="UserCancelled"/>
<ExitCode Value="1603" Result="Fail" String="GeneralFailure"/>
<ExitCode Value="3010" Result="SuccessReboot"/>
<DefaultExitCode Result="Fail" FormatMessageFromSystem="true" String="GeneralFailure" />
</ExitCodes>
</Command>
</Commands>
<!-- Defines a localizable string table for error messages-->
<Strings>
<String Name="DisplayName">.NET Framework 3.5 SP1</String>
<String Name="Culture">en</String>
<String Name="AdminRequired">Administrator permissions are required to install .NET Framework 3.5 SP1. Contact your administrator.</String>
<String Name="InvalidPlatformWinNT">Installation of the .NET Framework 3.5 SP1 requires Windows XP SP2, Windows 2003 SP1, Windows Vista, or later. Contact your application vendor.</String>
<String Name="InvalidPlatformArchitecture">This version of the .NET Framework 3.5 SP1 is not supported on an IA-64 operating system. Contact your application vendor.</String>
<String Name="UserCancelled">The user has cancelled the installation. .NET Framework 3.5 SP1 has not been installed.</String>
<String Name="GeneralFailure">A failure occurred attempting to install .NET Framework 3.5 SP1.</String>
<String Name="DotNetFX35SP1Exe">http://go.microsoft.com/fwlink/?linkid=118076</String>
</Strings>
Between the "InstallConditions" tags, you can add/remove conditions for prerequisite's install.Then between the "strings" tags, you can customize error messages.
For more info: http://msdn.microsoft.com/en-us/library/ms229223.aspx