I always use WinError.h. That has the vast majority of Windows error codes of all sorts.
A key indicator to look out for is the Facility part of the code: the second most-significant byte. That is, 0x80nnmmmm, where nn is the Facility. That tells you which component generated the code. Anything with a facility of 7 is a Windows error code repackaged as an HRESULT, and you should convert the low word to decimal and look it up in WinError.h. There are also error ranges that appear in their own headers (e.g. anything from 12000 - 12999 is a WinInet error code and you should look it up in WinInet.h).
Looking up the error code will give you the symbolic name, which might be found in more documentation than the code itself or the wording of the error message.
FACILITY_ITF (which has the value 4, so these HRESULTs start 0x8004) indicates that the error is defined by the interface you're using; you'll have to check with that interface to find out what it means.
Finally, COM also offers the interface IErrorInfo to retrieve extended error information: call GetErrorInfo to retrieve the error object. You'll have to query for ISupportErrorInfo and call that interface's InterfaceSupportsErrorInfo method to determine whether the interface you called actually set the error object (and of course, if it was template code, it could be lying).