tags:

views:

150

answers:

4

Related to a post of mine ( http://stackoverflow.com/questions/3122306/how-to-retrieve-a-file-from-internet-via-http ) about how to easily and robustly download a file from Internet, I have found a possible solution - however is not working as it was supposed to work.

According to MS documentation, the code below is supposed to time-out at 500ms after I disconnect myself from internet. However, it looks like it totally ignores the 'INTERNET_OPTION_RECEIVE_TIMEOUT' setting. The application freezes during download. It takes about 20-30 to this function to realize that there the Internet connection is down and to give the control back to the GUI.

Anybody knows why?

function GetBinFileHTTP (const aUrl: string; const pStream: TStream; wTimeOut: Word= 500; wSleep: Word= 500; wAttempts: Word= 10): Integer;
CONST
  BufferSize = 1024;
VAR
  hSession, hService: HINTERNET;
  Buffer     : array[0..BufferSize-1] of Char;
  dwBytesRead, dwBytesAvail: DWORD;
  lSucc        : LongBool;
  lRetries, dwTimeOut: Integer;   
begin
 Result:= 0;
 if NOT IsConnectedToInternet then
  begin
   Result:= -1;
   EXIT;
  end;

 hSession := InternetOpen(PChar(ExtractFileName(Application.ExeName)), INTERNET_OPEN_TYPE_PRECONFIG, nil, nil, 0);  { The INTERNET_OPEN_TYPE_PRECONFIG flag specifies that if the user has configured Internet Explorer to use a proxy server, WinInet will use it as well. }
 if NOT Assigned(hSession) then
  begin
   Result:= -4;
   EXIT;
  end;

 TRY
   hService := InternetOpenUrl(hSession, PChar(aUrl), nil, 0, INTERNET_FLAG_RELOAD, 0);
   if NOT Assigned(hService) then Exit;
   TRY
     FillChar(Buffer, SizeOf(Buffer), 0);

     { Set time out }
     dwTimeOut:= wTimeOut;
     InternetSetOption(hService, INTERNET_OPTION_RECEIVE_TIMEOUT, @dwTimeOut, SizeOf(dwTimeOut));   { use INTERNET_FLAG_RELOAD instead of NIL to redownload the file instead of using the cache }


     InternetSetOption(hService, INTERNET_OPTION_CONNECT_TIMEOUT, @dwTimeOut, SizeOf(dwTimeOut));

     REPEAT
       lRetries := 0;

       REPEAT
         lSucc:= InternetQueryDataAvailable( hService, dwBytesAvail, 0, 0);
         if NOT lSucc
         then Sleep( wSleep );
         if lRetries > wAttempts
         then Result:= -2;
       UNTIL lSucc OR (Result= -2);

       if NOT InternetReadFile(hService, @Buffer, BufferSize, dwBytesRead) then
        begin
          Result:= -3;                                                          { Error: File not found/File cannot be downloaded }
          EXIT;
        end;

       if dwBytesRead = 0
       then Break;

       pStream.WriteBuffer(Buffer[0], dwBytesRead);

     UNTIL False;
   FINALLY
     InternetCloseHandle(hService);
   end;
 FINALLY
   InternetCloseHandle(hSession);
 end;

 Result:= 1;
end;

Here is the documentation:

{

INTERNET_OPTION_CONNECT_TIMEOUT         Sets or retrieves an unsigned long integer value that contains the time-out value to use for Internet connection requests. If a connection request takes longer than this time-out value, the request is canceled. When attempting to connect to multiple IP addresses for a single host (a multihome host), the timeout limit is cumulative for all of the IP addresses. This option can be used on any HINTERNET handle, including a NULL handle. It is used by InternetQueryOption  and InternetSetOption.
INTERNET_OPTION_RECEIVE_TIMEOUT         Sets or retrieves an unsigned long integer value that contains the time-out value to receive a response to a request.      If the response takes longer than this time-out value, the request is canceled. This option can be used on any HINTERNET handle, including a NULL handle. It is used by InternetQueryOption and InternetSetOption. For using WinInet synchronously, only the default value for this flag can be changed by calling InternetSetOption and passing NULL in the hInternet parameter.
                  INTERNET_OPTION_CONTROL_RECEIVE_TIMEOUT - Identical to INTERNET_OPTION_RECEIVE_TIMEOUT. This is used by InternetQueryOption and InternetSetOption.
}

Edit: I disconnect the Internet by unplugging the cable or (for wireless) from software AFTER the application starts the download (I chose to download large file). It simulates the web site going offline.

Edit: How do I answer my own question? I have found the solution?

A: 

IMO, you should run this in a thread. Threading does not have to mean looping - it can be a "one and done" thread. Run it that way, and your GUI remains responsive until the thread finishes. I realize that this does not actually answer your question, but it will make your code better.

Also, if you disconnect the internet during the first loop where you're checking for data, I think it will retry 10 times. You should detect, and then quit right away.

Lastly, I don't think you should use EXIT when you've got handles and stuff open. Break instead, so that you still run through the disconnects. I would expect your code to tie up the socket. I saw this recently during a code review when there was an EXIT intead of a BREAK, and it's causing a memory leak because objects are created and never freed. I'd use the same rule here.

Chris Thornton
-1. Does not answer the question. Besides, how do you know this code isn't *already* running in a separate thread?
Rob Kennedy
The code inside `finally` will execute even if you `exit`.
Andreas Rejbrand
@Chris, I think that exit may be fine here since the OP is using try...finally to close the connections.
Marcus Adams
@Rob - he said his app freezes, so I assume that he's not running this in a thread.
Chris Thornton
No, I am not running it in a thread. I want to do that, but I am not sure how thread safe is that code.
Altar
@Andreas - does Exit really work like that? I assumed it just quit out of the function/procedure. If it does hit the finally, then I'm ok with that. I guess I've looked that Exit incorrectly for a long time then!
Chris Thornton
@Chris Thornton: Yes. Try it yourself! `try Exit; finally ShowMessage('test'); end;`
Andreas Rejbrand
@Altar, You can use a singleton thread even if it's not "thread safe".
Marcus Adams
A: 

Are you sure that you aren't hitting the INTERNET_OPTION_CONNECT_TIMEOUT? It will try to connect first, then receive.

In order to test the connect timeout, it must resolve, but never connect. In order to test the read timeout, it must connect, but never receive any data.

I generally set my connect timeout to 10 seconds, and the read timeout to 30 seconds. Anything longer than that, I consider down anyway.

Marcus Adams
"Are you sure that you aren't hitting the INTERNET_OPTION_CONNECT_TIMEOUT? It will try to connect first, then receive." - Absolutely sure!! I have done some tests for that.
Altar
+2  A: 

The connect timeout obviously isn't applicable in your test because by the time you start your test (i.e., pull the plug), the connection has already been established. Indeed, the connection is already established before you even get around to setting the timeout option.

The validity of the receive timeout is also suspect, because you've already begun receiving the response, too.

The most promising-looking timeout is the disconnect timeout, but MSDN says that's not implemented yet.

It seems to me that the way to go is to use asynchronous operations. Use InternetReadFileEx and use the irf_Async and irf_No_Wait flags. If too much time passes without receiving any data, close the connection. Another option is to stick with your synchronous calls, but then call InternetCloseHandle from another thread if the download takes too long.

Rob Kennedy
I disconnect the Internet by unplugging the cable or (for wireless) from software AFTER the application starts the download (I chose to download large file). It simulates the web site going offline.
Altar
Right. If the download has started, then the connection has already been established and the response has already begun. Therefore, the connection timeout is no longer relevant, and neither is the receipt timeout.
Rob Kennedy
So, basically there is no time-out implemented in that library (after the download starts).
Altar
+1  A: 

There is a documented bug in MS IE code. Can only be solved by using the code in a thread and re-implementing the time out mechanism.

Details:

"This acticle shows a workaround to the InternetSetOption API bug on setting timeout values by creating a second thread. InternetSetOption Does Not Set Timeout Values"

http://support.microsoft.com/default.aspx?scid=kb;en-us;Q224318

Maybe somebody can help with implementing this bug fix also in Delphi. I personally don't have experience with C. Even the backbone in pseudo-Pascal will be nice.

Altar
Got a link to that documentation?
Rob Kennedy
Hi Rob, see the link I just added.
Altar
Looks similar to the last suggestion in my answer. Instead of waiting for the timeout and calling InternetCloseHandle in the separate thread, though, the KB article sets the connection in the other thread. The example code is almost entirely API calls. Which part is giving you trouble?
Rob Kennedy