C# .NET HttpWebClient can't get file upload working; troubleshooting hints?

Question

Hi,

I need to use HttpWebClient to upload a series of files at once.

I am following the sample code found on this thread:

http://stackoverflow.com/questions/566462/upload-files-with-httpwebrequest-multipart-form-data

As several people reported on that thread, I am also getting HTTP status 500 errors. I have tried 2 of the solutions offered in the thread, but neither one works for me. It seems that there is some problem connected with spacing in the request, but I haven't been able to figure it out.

Questions:

1. What resource should I used to verify that the HTTP request is formatted correctly?

2. It should be possible to manually write the request if I copy and paste the file section, right?

3. If you know of some sample code that works, please let me know.

Here is the code that I am using:

 public static  void UploadFilesToRemoteUrl(string url, string[] files,
 string logpath, NameValueCollection nvc)
 {

long length = 0;
string boundary = "----------------------------" +
DateTime.Now.Ticks.ToString("x");


HttpWebRequest httpWebRequest2 = (HttpWebRequest)WebRequest.Create(url);
httpWebRequest2.ContentType = "multipart/form-data; boundary=" +
boundary;
httpWebRequest2.Method = "POST";
httpWebRequest2.KeepAlive = true;
httpWebRequest2.Credentials =
System.Net.CredentialCache.DefaultCredentials;



Stream memStream = new System.IO.MemoryStream();

byte[] boundarybytes = System.Text.Encoding.ASCII.GetBytes("\r\n--" +
boundary + "\r\n");


string formdataTemplate = "\r\n--" + boundary +
"\r\nContent-Disposition: form-data; name=\"{0}\";\r\n\r\n{1}";

foreach (string key in nvc.Keys)
{
    string formitem = string.Format(formdataTemplate, key, nvc[key]);
    byte[] formitembytes = System.Text.Encoding.UTF8.GetBytes(formitem);
    memStream.Write(formitembytes, 0, formitembytes.Length);
}


memStream.Write(boundarybytes, 0, boundarybytes.Length);

string headerTemplate = "Content-Disposition: form-data; name=\"{0}\"; filename=\"{1}\"\r\n Content-Type: application/octet-stream\r\n\r\n";

for (int i = 0; i < files.Length; i++)
{

    //string header = string.Format(headerTemplate, "file" + i, files[i]);
    string header = string.Format(headerTemplate, "uplTheFile", files[i]);

    byte[] headerbytes = System.Text.Encoding.UTF8.GetBytes(header);

    memStream.Write(headerbytes, 0, headerbytes.Length);


    FileStream fileStream = new FileStream(files[i], FileMode.Open,
    FileAccess.Read);
    byte[] buffer = new byte[1024];

    int bytesRead = 0;

    while ((bytesRead = fileStream.Read(buffer, 0, buffer.Length)) != 0)
    {
        memStream.Write(buffer, 0, bytesRead);

    }


    memStream.Write(boundarybytes, 0, boundarybytes.Length);


    fileStream.Close();
}

httpWebRequest2.ContentLength = memStream.Length;

Stream requestStream = httpWebRequest2.GetRequestStream();

memStream.Position = 0;
byte[] tempBuffer = new byte[memStream.Length];
memStream.Read(tempBuffer, 0, tempBuffer.Length);
memStream.Close();
requestStream.Write(tempBuffer, 0, tempBuffer.Length);
requestStream.Close();


WebResponse webResponse2 = httpWebRequest2.GetResponse();

Stream stream2 = webResponse2.GetResponseStream();
StreamReader reader2 = new StreamReader(stream2);


MessageBox.Show(reader2.ReadToEnd());

webResponse2.Close();
httpWebRequest2 = null;
webResponse2 = null;
}

Update:

Thanks to ck for encouraging me to keep using fiddler. That was the direction that I needed to find the solution to this problem.

Now let me answer the questions here.

1. What resource should I used to verify that the HTTP request is formatted correctly? a) The standards. There is RFC1867 http://www.faqs.org/rfcs/rfc1867.html . Nice read. It explains who the multiple upload is supposed to happen, and also how you handle the other form data. The limitation is that it doesn't outright say how the request should be formatted in terms of number of CRLF between each header and values. I suspect that the information may be in the HTTP RFC 2616 http://www.w3.org/Protocols/rfc2616/rfc2616.html but that is a bit longer reading for the time I had. Also, there may be some room for interpretation between different implementations of it.

b) Actual Examples. The winning combination for getting real examples were using curl and fiddler together. I was able to send the correct uploading request via curl, but curl's --trace-ascii wasn't giving me the raw header that I needed. The approximations that I made based on it kept failing, and fiddler didn't seem to capture its traffic.

Then I found a link that explain that curl doesn't use the system's proxy. So to capture its traffic on fiddler, one must enter curl -x 127.0.0.1:8888 www.happysmurfs.com Now I got the raw header that I needed. NOW, with the correct raw header, troubleshooting the code was easy, since I had another text header to compare it with it.

2. It should be possible to manually write the request if I copy and paste the file section, right? Yes, it seems that it is possible, but as ck mentioned, it is a lot easier to get a working, proven header from fiddler and then make modifications from there.

3. If you know of some sample code that works, please let me know. The code above should work, but watch out for this line:

 string formdataTemplate = "\r\n--" + boundary +
"\r\nContent-Disposition: form-data; name=\"{0}\";\r\n\r\n{1}";

The first \r\n shouldn't be there. Also, there shouldn't be no double new lines, (\r\n), at then end of headers or boundaries.

Answer 1

You should use Fiddler to verify what your program is sending is the same as what you would send if doing it manually.