BITS client fails to specify HTTP Range header

Question

Our system is designed to deploy to regions with unreliable and/or insufficient network connections. We build our own fault tolerating data replication services that uses BITS.

Due to some security and maintenance requirements, we implemented our own ASP.NET file download service on the server side, instead of just letting IIS serving up the files. When BITS client makes an HTTP download request with the specified range of the file, our ASP.NET page pulls the demanded file segment into memory and serve that up as the HTTP response. That is the theory. ;) This theory fails in artificial lab scenarios but I would not let the system deploy in real life scenarios unless we can overcome that.

Lab scenario: I have BITS client and the IIS on the same developer machine, so practically I have enormous network "bandwidth" and BITS is intelligent enough to detect that. As BITS client discovers the unlimited bandwidth, it gets more and more "greedy". At each HTTP request, BITS wants to grasp greater and greater file ranges (we are talking about downloading CD iso files, videos), demanding 20-40MB inside a single HTTP request, a size that I am not comfortable to pull into memory on the server side as one go. I can overcome that simply by giving less than demanded. It is OK.

However, BITS gets really "confident" and "arrogant" demanding files WITHOUT specifying the download range, i.e., it wants the entire file in a single request, and this is where things go wrong. I do not know how to answer that response in the case of a 600MB file. If I just provide the starting 1MB range of the file, BITS client keeps sending HTTP requests for the same file without download range to continue, it hammers its point that it wants the entire file in one go. Since I am reluctant to provide the entire file, BITS gives up after several trials and reports error.

Any thoughts?

Answer 1

It seems, we could solve the issue. There are several things:

• We needed to have control over every incoming request, so we could not just let IIS handle the request alone. However, it is still not necessary pull the files into memory just to facilitate the download: Request.TransferFile(...) actually does all the heavy lifting without pulling the files into memory, while we still maintain the control over handling the request.
• BITS implementation looks very intelligent how to handle various cases. We learned if BITS requests the full file without range specified, even if it is several GB, we can just let it have it. As soon as it receives the first HTTP packets and figures out that this is bigger than it could swallow, it cancels the request immediately and re-requests it with appropriate range.
• BITS allows to configure maximum bandwidth and to schedule when it is allowed to use network resources. While this configuration is applicable to the client side, it is OK in our particular case.