Need loop to copy chunks from byte array

Question

I have to process a large byte array that is passed to my function. I need to copy the content from this incoming byte array in smaller "chunks" to an outbound byte array.

For every "chunk" of data created in the outbound array, I need to call a web service.

Upon return, I need to resume looping through the incoming byte array, continuing to pass a whole or partial chunk of data until the complete incoming array is processed (i.e. sent to the web service in chunks).

I am very new to C# and I am struggling with a loop that works. I know how to call the web service to handle a "chunk" but I can't get the looping correct. Here is a sketch of the pathetic mess I currently have:

int chunkSize = 10000;
byte[] outboundBuffer = new byte[chunkSize];     
while (BytesRead > 0)
{
long i = 0;
foreach (byte x in incomingArray)
{
    BytesRead += 1;
    outboundBuffer[i] = incomingArray[i]
    i++;
}
uploadObject.Size = BytesRead;
uploadObject.MTOMPayload = outboundBuffer;

// call web service here and pass the uploadObject 

// get next "chunk" until incomingArray is fully processed 
 }

I know this is a mess and won't work; could someone sketch a proper loop to get this done? Thanks very much.

Answer 1

You might want to look into Array.Copy or Buffer.BlockCopy; this will clean things up a bit, since you won't have to copy all of the bytes individually:

int incomingOffset = 0;

while(incomingOffset < incomingArray.Length)
{
   int length = 
      Math.Min(outboundBuffer.Length, incomingArray.Length - incomingOffset);

   // Changed from Array.Copy as per Marc's suggestion
   Buffer.BlockCopy(incomingArray, incomingOffset, 
                    outboundBuffer, 0, 
                    length);

   incomingOffset += length;

   // Transmit outbound buffer
 }

Answer 2

You seem to have broken down your task logically, after all, you've coherently described it with words. Now just make your code do it.

Pseudo code might be something like:

while (there are chunks in incoming array)
    copy 1 chunk to outbound array
    create uploadObject
    call webservice
endwhile

Answer 3

You probably want Buffer.BlockCopy (the rawest of the copies; ideally suitable for byte[]).

Of course, the other option is to use a MemoryStream in place of the outbound array, and just Write to it each time, then call ToArray() or GetBuffer() on the MemoryStream (with GetBuffer(), you need to watch the length; with ToArray() it is trimmed for you automatically):

MemoryStream ms = new MemoryStream();
byte[] buffer = new byte[BUFFER_SIZE];
int bytesReceived;
while((bytesReceived = GetNextChunk(buffer, 0, BUFFER_SIZE)) > 0) {
    ms.Write(incomingArray, 0, bytesReceived);
}
byte[] final = ms.ToArray();

Answer 4

Be wary of calling web-services in a loop synchronously. Synchronous web-service calls take indefinite time due to the nature of HTTP and your loop might run for a long time. It is preferable to use an asynchronous approach.

Answer 5

Why not just use Array.Copy? http://msdn.microsoft.com/en-us/library/system.array.copy.aspx

eg.

Array.Copy(srcBuffer, destBuffer, 1024);