On my application wide error handling I'm returning the host name to see what computer it happened on, application version, windows version, exception with all inner exceptions, and the stack trace.
What other information can you get while inside the MyApplication_UnhandledException method that would be useful for tracking down an error?
If you use something like log4net, you can get thread information, which has proved to be useful on more than one occasion for tracking down an elusive bug.
As an example, you might want to look at what ASP.NET Health Monitoring places in the event log when it processes an unhandled exception. Also look at Enterprise Library to see what the Logging Application Block permits you to add.