HttpSendRequest not getting latest file from server

Question

I am having an issue with my HTTP requests in my app, such that if the remote file is the same size as the local file (even though its modified time is different, as its contents have been changed), attempts to download it return quickly and the newer file is not downloaded.

In short, the process I am following is: Setting up an HTTP connection with the INTERNET_FLAG_RESYNCHRONIZE flag and calling HttpSendRequest(); then checking the HTTP status code and finding it to be "200".

• If the remote file is updated, but remains the same size as the local copy: The local file is unchanged after running the app. If I call HttpQueryInfo() with HTTP_QUERY_LAST_MODIFIED after sending the request, it gives me the actual last modified time of the server's file, which I can see is different from the local file I am trying to have it overwrite.
• If the remote file is updated, and the file size becomes different from the local copy: It is downloaded and overwrites the local copy as expected.

Here's a fairly abridged version of the code, to cut out helpers and error checking:

// szAppName = our app name
HINTERNET hInternetHandle = InternetOpen( szAppName,
    INTERNET_OPEN_TYPE_PRECONFIG, NULL, NULL, 0 );
// szServerName = our server name
hInternetHandle = InternetConnect( hInternetHandle, szServerName,
    INTERNET_DEFAULT_HTTP_PORT, NULL, NULL, INTERNET_SERVICE_HTTP, NULL, 0 );

// szPath = the file to download
LPCSTR aszDefault[2] = { "*/*", NULL };
DWORD dwFlags = 0
| INTERNET_FLAG_IGNORE_REDIRECT_TO_HTTP
| INTERNET_FLAG_IGNORE_REDIRECT_TO_HTTPS
| INTERNET_FLAG_KEEP_CONNECTION
| INTERNET_FLAG_NO_AUTH
| INTERNET_FLAG_NO_AUTO_REDIRECT
| INTERNET_FLAG_NO_COOKIES
| INTERNET_FLAG_NO_UI
| INTERNET_FLAG_RESYNCHRONIZE;
HINTERNET hHandle = HttpOpenRequest( hInternetHandle, "GET", szPath, NULL,
    NULL, aszDefault, dwFlags, 0 );

DWORD dwTimeOut = 10 * 1000; // In milliseconds
InternetSetOption( hInternetHandle, INTERNET_OPTION_CONNECT_TIMEOUT,
    &dwTimeOut, sizeof( dwTimeOut ) );
InternetSetOption( hInternetHandle, INTERNET_OPTION_RECEIVE_TIMEOUT,
    &dwTimeOut, sizeof( dwTimeOut ) );
InternetSetOption( hInternetHandle, INTERNET_OPTION_SEND_TIMEOUT,
    &dwTimeOut, sizeof( dwTimeOut ) );
DWORD dwRetries = 5;
InternetSetOption( hInternetHandle, INTERNET_OPTION_CONNECT_RETRIES,
    &dwRetries, sizeof( dwRetries ) );

HttpSendRequest( hInternetHandle, NULL, 0, NULL, 0 );

Since I have found I can query the remote file's last modified time, and find it to be accurate, I know it's actually getting to the server. I thought that specifying INTERNET_FLAG_RESYNCHRONIZE would force the file to resynch if it's out of date. Do I have it all wrong? Is this just how it's supposed to work?

---

Edit: I've done some investigation with a packet sniffer, and here's some additional information:

If the remote and local file are exactly the same, this is the exchange:

GET /test.bmp HTTP/1.1
Accept: */*
If-None-Match: "1c1467112ee6ca1:369"
User-Agent: Internal Testing
Host: ****************
Connection: Keep-Alive

HTTP/1.1 304 Not Modified
Last-Modified: Tue, 27 Apr 2010 17:21:26 GMT
Accept-Ranges: bytes
ETag: "1c1467112ee6ca1:369"
Server: Microsoft-IIS/6.0
X-Powered-By: ASP.NET
Date: Tue, 27 Apr 2010 18:10:26 GMT

Now, if the remote file has changed, but remains the same file size:

GET /test.bmp HTTP/1.1
Accept: */*
If-None-Match: "1c1467112ee6ca1:369"
User-Agent: Internal Testing
Host: ****************
Connection: Keep-Alive

HTTP/1.1 200 OK
Content-Length: 419958
Content-Type: image/bmp
Last-Modified: Tue, 27 Apr 2010 18:11:17 GMT
Accept-Ranges: bytes
ETag: "b65425835e6ca1:369"
Server: Microsoft-IIS/6.0
X-Powered-By: ASP.NET
Date: Tue, 27 Apr 2010 18:11:33 GMT
[Block of data]

So, the server is indeed sending the file when it has changed, but my app still seems to see it as not having changed. I am thinking the problem lies in how my app is dealing with the response; it's not my own code, and the guy who wrote it has moved on long ago.

One problem I've found is that in both of the above scenarios, when I call HttpQueryInfo() with HTTP_QUERY_STATUS_CODE I get 200 back. However, in the first case above I can see that the actual server response was 304, not 200. Digging into the code we're using, I found that it seems to be trying to work around this by doing a filesize comparison, and assuming the file hasn't changed if the filesizes are the same; hence, the problem I am having!

So now my question is more simply: why would HttpQueryInfo() return 200 even when the server is returning one of the 3XX errors? I've found some people asking a similar question online, but they either received no responses or were dealing with web browsers directly.

Answer 1

I believe that INTERNET_FLAG_RESYNCHRONIZE uses the If-Modified-Since HTTP request header field and checks the status code while HTTP_QUERY_LAST_MODIFIED will (I think) just does a HEAD request & checks the HTTP response Last-Modified header field.

Try a HTTP_QUERY_IF_UNMODIFIED_SINCE & a HTTP_QUERY_LAST_MODIFIED with HttpQueryInfo() and compare the results [as a side note if you have a HTTP sniffer up & running it may clear some things up for all of us...].

As a quick & dirty solution you can use the INTERNET_FLAG_RELOAD instead of INTERNET_FLAG_RESYNCHRONIZE to force the file reload each & every time you request it.

HTH