views:

378

answers:

4

Here's the setup:

  • 1 web server running a C# app to which my users (stored in a MySQL database on said server) authenticate.

  • 1 file server running software TBD. In the past I've used lighttpd and mod_secdownload to secure the files on the file servers, and it's worked well(ish).

I'm wondering if there is a way to do this using a combination of IIS and C# .Net. All my other servers are running that combo, and it would simplify things a bit if I could do the same for the file servers. The kicker is, the files that are being hosted are large. I've seen examples of people using a small app to create a FileStream object, read in the file, and create the HTTP Response by hand. This works, but since I'm working with files 500+ MB in size, it's slow as heck. And I'll potentially have 300 users hitting the box at once, requesting files. That's no good.

So, anyone see a way around this? I'm trying to create a more transparent system, and if all my servers are running the same software/hardware, it will make my life a whole lot simpler. Thanks in advance for any advice you give!

+6  A: 

You know what? The KB article is poo. Here is my official recommendation:

public void StreamFile(string filePath)
{
    string fileName = Path.GetFileName(filePath);

    using (var fStream = new FileStream(filePath, FileMode.Open, FileAccess.Read, FileShare.Read))
    {
        var contentLength = fStream.Length;

        if (Request.UserAgent.Contains("MSIE"))
        {
            Response.AddHeader("Content-Transfer-Encoding", "binary");
        }

        Response.ContentType = "application/octet-stream";
        Response.AddHeader("Content-Length", contentLength.ToString());

        // Even though "Content-Disposition" should have an upper-case "d", as per http://www.ietf.org/rfc/rfc2183.txt
        // IE fails to recognize this if the "d" is upper-cased.
        Response.AddHeader("Content-disposition", "attachment; filename=" + fileName);

        var buffer = new byte[8192];

        while (Response.IsClientConnected)
        {
            var count = fStream.Read(buffer, 0, buffer.Length);
            if (count == 0)
            {
                break;
            }

            Response.OutputStream.Write(buffer, 0, count);
            Response.Flush();
        }
    }

    Response.Close();
}
John Gietzen
After giving that an attempt, it looks to me like that still loads the file into memory. At least, I was getting out of memory exceptions when I tried it on a 1.5GB file, and got tired of waiting for the download on a 450MB one. I'm doing this on a test machine, not the file server, so the amount of memory I have to work with is lower.My file servers have a ton of ram (24GB I think) but I'd like a solution that doesn't require a lot of strain on the processor or memory. Lighttpd does this using mod_secdownload and URL rewrite rules, which is fast and cheap on cycles/memory.
Sheep Slapper
@john - the way I read the code in Reflector, none of the MS methods for writing files actually use small buffers. They all seem to allocate a single buffer for the entire file. Some of this is pretty hard to trace, so maybe I am missing something.
Ray
@Sheep Slapper: http://support.microsoft.com/kb/812406
sixlettervariables
@Ray, crap. Well, I'll update with the KB Article Info
John Gietzen
@John: You beat me to it. I just tested that, and it worked. If there is a solution that doesn't require me to create HTTP streams by hand, and take up memory, that would be preferred. Lighttpd does it, and does it well, but I'm trying to keep the number of technologies involved to a minimum.For now, I guess this works. If anyone comes across a way to do this without the ugly FileStream code, that would rock. But for now, John's is the accepted answer.
Sheep Slapper
@Sheep: Please don't use the code directly from the KB article of from Ray's link. Both of those are very messy versions of the code I have above. Also, I have a bug fix for one of IE's quirks.
John Gietzen
@Sheep Slapper: keep in mind the way ASP.Net works you can't make it as nice as lighttpd. The worker process sends data to ISAPI that sends it to the user. Two places with buffering is what you'll have to live with.
sixlettervariables
Also may I suggest a buffer larger than 1024? Maybe 4k or 64k?
sixlettervariables
@sixletter: Yeah, but he was trying to stay low on the memory usage. I'm pretty sure that 1024 bytes would be sufficient to peg the bandwidth.
John Gietzen
1k in a loop with a flush is pretty inefficient.
sixlettervariables
@six: Good point. Changed to 8k.
John Gietzen
A: 

You may be interested in Microsoft's Background Intelligent Transfer Service (BITS).

http://msdn.microsoft.com/en-us/library/bb968799%28VS.85%29.aspx

Version 2.5 introduced HTTP authentication via certificates.

gooch
+1  A: 

This thread has my solution to keep memory usage down while users are downloading files. You probably want a bigger buffer than my sample uses, though.

Ray
After reading that thread and the Microsoft Support article in the answer above, I'm headed in that direction for a temporary solution. It's better in that there are less applications involved in the process, but it's not as transparent in what's going on. Now I just have to develop a tiny app to implement it and build my own version of mod_secdownload since the app that validates my users isn't on the file servers! Thanks for the info!
Sheep Slapper
A: 

While this isn't directly applicable since you're using MySql but it's something to consider (might even be available for free by using sql server 2008 express). Sql Server 2008 offers Filestream support which lets you access files as if they are directly stored in the database but actually reside on a fileserver. Then information on the other posts can help you with getting it to the user.

FILESTREAM Storage in SQL Server 2008

Chris Marisic