Calculate upload transfer speed problem

Question

Hey guys,

I have implemented a file transfer rate calculator to display kB/sec for an upload process occuring in my app, however with the following code it seems I am getting 'bursts' in my KB/s readings just after the file commences to upload.

This is the portion of my stream code, this streams a file in 1024 chunks to a server using httpWebRequest:

using (Stream httpWebRequestStream = httpWebRequest.GetRequestStream())
{
    if (request.DataStream != null)
    {
        byte[] buffer = new byte[1024];
        int bytesRead = 0;

        Debug.WriteLine("File Start");
        var duration = new Stopwatch();
        duration.Start();
        while (true)
        {
            bytesRead = request.DataStream.Read(buffer, 0, buffer.Length);
            if (bytesRead == 0)
                break;

            httpWebRequestStream.Write(buffer, 0, bytesRead);
            totalBytes += bytesRead;


            double bytesPerSecond = 0;
            if (duration.Elapsed.TotalSeconds > 0)
                bytesPerSecond = (totalBytes / duration.Elapsed.TotalSeconds);

            Debug.WriteLine(((long)bytesPerSecond).FormatAsFileSize());
        }
        duration.Stop();
        Debug.WriteLine("File End");
        request.DataStream.Close();
    }
}

Now an output log of the upload process and associated kB/sec readings are as follows: (You will note a new file starts and ends with 'File Start' and 'File End')

File Start
5.19 MB
7.89 MB
9.35 MB
11.12 MB
12.2 MB
13.13 MB
13.84 MB
14.42 MB
41.97 kB
37.44 kB
41.17 kB
37.68 kB
40.81 kB
40.21 kB
33.8 kB
34.68 kB
33.34 kB
35.3 kB
33.92 kB
35.7 kB
34.36 kB
35.99 kB
34.7 kB
34.85 kB
File End

File Start
11.32 MB
14.7 MB
15.98 MB
17.82 MB
18.02 MB
18.88 MB
18.93 MB
19.44 MB
40.76 kB
36.53 kB
40.17 kB
36.99 kB
40.07 kB
37.27 kB
39.92 kB
37.44 kB
39.77 kB
36.49 kB
34.81 kB
36.63 kB
35.15 kB
36.82 kB
35.51 kB
37.04 kB
35.71 kB
37.13 kB
34.66 kB
33.6 kB
34.8 kB
33.96 kB
35.09 kB
34.1 kB
35.17 kB
34.34 kB
35.35 kB
34.28 kB
File End

My problem is as you will notice, the 'burst' I am talking about starts at the beginning of every new file, peaking in MB's and then evens out properly. is this normal for an upload to burst like this? My upload speeds typically won't go higher than 40k/sec here so it can't be right.

This is a real issue, when I take an average of the last 5 - 10 seconds for on-screen display, it really throws things out producing a result around ~3MB/sec!

Any ideas if I am approaching this problem the best way? and what I should do? :S

Graham

Also: Why can't I do 'bytesPerSecond = (bytesRead / duration.Elapsed.TotalSeconds)' and move duration.Start & duration.Stop into the while loop and receive accurate results? I would have thought this would be more accurate? Each speed reads as 900 bytes/sec, 800 bytes/sec etc.

Answer 1

The way i do this is: Save up all bytes transfered in a long.

Then every 1 second i check how much has been transfered. So i basicly only trigger the code to save speed once pr second. Your while loop is going to loop maaaaaaaaaaaany times in one second on a fast network.

Depending on the speed of your network you may need to check the bytes transfered in a seperate thread or function. I prefere doing this with a Timer so i can easly update UI

EDIT: From your looking at your code, im guessing what your doing wrong is that you dont take into account that one loop in the while(true) is not 1 second

EDIT2: Another advatage with only doing the speed check once pr second is that things will go much quicker. In cases like this updating the UI can be the slowest thing your are doing, so if you try to update the UI every loop, thats most likely your slowest point and is going to produce unresponsive UI.

Your also correct that you should avarage out the values, so you dont get the microsoft minutes bugs. I normaly do this in the Timer function running by doing something like this:

//Global variables
long gTotalDownloadedBytes;
long gCurrentDownloaded; // Where you add up from the download/upload untill the speedcheck is done.
int gTotalDownloadSpeedChecks;


//Inside function that does speedcheck    
gTotalDownloadedBytes += gCurrentDownloaded;
gTotalDownloadSpeedChecks++;

long AvgDwnSpeed = gTotalDownloadedBytes / gTotalDownloadSpeedChecks; // Assumes 1 speedcheck pr second.

Answer 2

All, I think I have fixed my issue by adjusting the 5 - 10 averging variable to wait one second to account for the burst, not the best, but will allow internet to sort itself out and allow me to capture a smooth transfer.

It appears from my network traffic it down right is bursting so there is nothing in code I could do differently to stop this.

Please will still be interested in more answers before I hesitantly accept my own.

Answer 3

There's many layers of software and hardware between you and the system you're sending to, and several of those layers have a certain amount of buffer space available.

When you first start sending, you can pump out data quite quickly until you fill those buffers - it's not actually getting all the way to the other end that fast, though! After you fill up the send buffers, you're limited to putting more data into them at the same rate it's draining out, so the rate you see will drop to the underlying networking sending rate.