Which reasons could make ShellExecute fail?

Question

I have a VB6 application which opens files with their associated application using:

ShellExecute(0, "open", filename, params, vbNullString, vbNormalFocus)

This works perfectly.

Now I got a customer (running XP with Adobe Reader) who can't open any PDF file using the above command. But the same file is being opened without any problems when double clicking it from Windows Explorer. I also tested the filename/-path combination on my machine to exclude those kind of problems.

I'm searching for any hints on what I could check to make sure ShellExecute is working. Or what can cause ShellExecute to fail this way?

Answer 1

What's the return value of ShellExecute? If it's 0x0000001f (== 31, meaning SE_ERR_NOASSOC), than according to shellapi.h "There is no application associated with the given file name extension.", which means that somehow the registration of the .pdf file extension got lost. Reinstalling Adobe Reader might help.

Answer 2

Have a look at the return value of your ShellExecute call. From the MSDN:

If the function succeeds, it returns a value greater than 32. If the function fails, it returns an error value that indicates the cause of the failure. The return value is cast as an HINSTANCE for backward compatibility with 16-bit Windows applications. It is not a true HINSTANCE, however. It can be cast only to an int and compared to either 32 or the following error codes below.

0: The operating system is out of memory or resources.

ERROR_FILE_NOT_FOUND: The specified file was not found.

ERROR_PATH_NOT_FOUND: The specified path was not found

(...)

Answer 3

Further to Thomas's answer, here's some VB6 constants for possible return values of ShellExecute, with possible explanations (I think I originally took these from the MSDN page, return value section). A return value of 32 or less means the call failed. The specific value returned indicates what went wrong.

Const ERROR_BAD_FORMAT = 11&
Const ERROR_FILE_NOT_FOUND = 2&          
Const ERROR_PATH_NOT_FOUND = 3&          ' The specified path was not found. '
Const SE_ERR_ACCESSDENIED = 5            ' The operating system denied access to the specified file. '
Const SE_ERR_ASSOCINCOMPLETE = 27        ' The file name association is incomplete or invalid. '
Const SE_ERR_DDEBUSY = 30                ' The Dynamic Data Exchange (DDE) transaction could not be completed because other DDE transactions were being processed. '
Const SE_ERR_DDEFAIL = 29                ' The DDE transaction failed. '
Const SE_ERR_DDETIMEOUT = 28             ' The DDE transaction could not be completed because the request timed out. '
Const SE_ERR_DLLNOTFOUND = 32            ' The specified dynamic-link library (DLL) was not found. '
Const SE_ERR_FNF = 2                     ' The specified file was not found. '
Const SE_ERR_NOASSOC = 31                ' There is no application associated with the given file name extension. '
Const SE_ERR_OOM = 8                     '  out of memory '
Const SE_ERR_PNF = 3                     '  path not found '
Const SE_ERR_SHARE = 26                  ' A sharing violation occurred. '

Answer 4

You have "open" as the verb, don't do that, use vbNullString as the verb ("Open" means the open verb, NULL means the default verb (If the user has not set a specific default, the default is open, if there is no open verb for that filetype, ShellExecute uses the first verb it finds))

Answer 5

Instead of using ShellExecute to 'execute' the PDF file, I use the FindExecutable API:

Private Const ERROR_FILE_NO_ASSOCIATION     As Long = 31
Private Const ERROR_FILE_NOT_FOUND          As Long = 2
Private Const ERROR_PATH_NOT_FOUND          As Long = 3
Private Const ERROR_FILE_SUCCESS            As Long = 32 
Private Const ERROR_BAD_FORMAT              As Long = 11

Private Declare Function FindExecutable Lib "shell32.dll" _
   Alias "FindExecutableA" _
  (ByVal lpFile As String, _
   ByVal lpDirectory As String, _
   ByVal sResult As String) As Long


Private Sub OpenDocument(sFile as string, sPath as string)
     Dim sResult As String
     Dim lSuccess As Long, lPos as long

     sResult = Space$(MAX_PATH)
     lSuccess = FindExecutable(sFile, sPath), sResult)
     Select Case lSuccess
        Case ERROR_FILE_NO_ASSOCIATION
            If Right$(sFile, 3) = "pdf" Then
                MsgBox "You must have a PDF viewer such as Acrobat Reader to view pdf files."
            Else
                MsgBox "There is no registered program to open the selected file." & vbCrLf & sFile
            End If
        Case ERROR_FILE_NOT_FOUND: MsgBox "File not found: " & sFile
        Case ERROR_PATH_NOT_FOUND: MsgBox "Path not found: " & sPath
        Case ERROR_BAD_FORMAT:     MsgBox "Bad format."
        Case Is >= ERROR_FILE_SUCCESS:
           lPos = InStr(sResult, Chr$(0))
           If lPos Then sResult = Left$(sResult, lPos - 1)
           Shell sResult & " " & sPath & sFile, True), vbMaximizedFocus
    End Select

End Sub

Answer 6

uninstall and install acrobat reader, under documents and settings, rename "username" folder to "usernamex" (you should be logged in with different admin user), then relogin user and it creates new "username" folder with new user registry, now it should work.. you can copy user's files from usernamex folder to new username folder (desktop, documents etc)..

Answer 7

Here's a function that translates a the windows error numbers to text. You can use the return value as the parameter and get back a more friendly message.

Private Declare Function FormatMessage Lib "kernel32" Alias "FormatMessageA" _
    (ByVal dwFlags As Long, lpSource As Long, ByVal dwMessageId As Long, _
    ByVal dwLanguageId As Long, ByVal lpBuffer As String, _
    ByVal nSize As Long, ByVal Arguments As Any) As Long

Private Const FORMAT_MESSAGE_FROM_SYSTEM = &H1000
Private Const FORMAT_MESSAGE_IGNORE_INSERTS = &H200
Private Const MAX_PATH = 260

Function TranslateDLLError(ByVal lngErrNum As Long) As String
   Dim sRtrnCode As String * MAX_PATH
   Dim lRet As Long

   On Error GoTo errTranslateDLLError(

   sRtrnCode = Space$(256)
   lRet = FormatMessage(FORMAT_MESSAGE_FROM_SYSTEM Or FORMAT_MESSAGE_IGNORE_INSERTS, 0&, lngErrNum, 0&, sRtrnCode, Len(sRtrnCode), 0&)
   If lRet > 0 Then
      Translate_DLL_Error = Replace$(Left(sRtrnCode, lRet), vbCrLf, "")
   Else
      Translate_DLL_Error = "Error not found."
   End If

   Exit Function

errTranslateDLLError(:
   TranslateDLLError( = "Unable to translate system error: " & CStr(lngErrNum)

End Function

Answer 8

If you're using

CoInitializeEx(NULL, COINIT_MULTITHREADED)

in your code, then you will have to create a separate thread for executing via ShellExecute. See more here: Calling Shell Functions and Interfaces from a Multithreaded Apartment