C# byte[] substring? (design)

Question

I'm downloading some files asynchronously into a large byte array, and I have a callback that fires off periodically whenever some data is added to that array. If I want to give developers the ability to use the last chunk of data that was added to array, then... well how would I do that? In C++ I could give them a pointer to somewhere in the middle, and then perhaps tell them the number of bytes that were added in the last operation so they at least know the chunk they should be looking at... I don't really want to give them a 2nd copy of that data, that's just wasteful.

I'm just thinking if people want to process this data before the file has completed downloading. Would anyone actually want to do that? Or is it a useless feature anyway? I already have a callback for when the buffer (entire byte array) is full, and then they can dump the whole thing without worrying about start and end points...

Answer 1

I think you shouldn't bother.

Why on earth would anyone want to use it?

Answer 2

You can't give them a pointer into the array, but you could give them the array and start index and length of the new data.

But I have to wonder what someone would use this for. Is this a known need? or are you just guessing that someone might want this someday. And If so, is there any reason why you couldn't wait to add the capability once somone actually needs it?

Answer 3

Copying a chunk of a byte array may seem "wasteful," but then again, object-oriented languages like C# tend to be a little more wasteful than procedural languages anyway. A few extra CPU cycles and a little extra memory consumption can greatly reduce complexity and increase flexibility in the development process. In fact, copying bytes to a new location in memory to me sounds like good design, as opposed to the pointer approach which will give other classes access to private data.

But if you do want to use pointers, C# does support them. Here is a decent-looking tutorial. The author is correct when he states, "...pointers are only really needed in C# where execution speed is highly important."

Answer 4

That sounds like you want an event.

public class ArrayChangedEventArgs : EventArgs {
    public (byte[] array, int start, int length) {
        Array = array;
        Start = start;
        Length = length;
    }
    public byte[] Array { get; private set; }
    public int Start { get; private set; }
    public int Length { get; private set; }
}

// ...
// and in your class:

public event EventHandler<ArrayChangedEventArgs> ArrayChanged;

protected virtual void OnArrayChanged(ArrayChangedEventArgs e)
{
    // using a temporary variable avoids a common potential multithreading issue
    // where the multicast delegate changes midstream.
    // Best practice is to grab a copy first, then test for null

    EventHandler<ArrayChangedEventArgs> handler = ArrayChanged;

    if (handler != null)
    {
        handler(this, e);
    }
}

// finally, your code that downloads a chunk just needs to call OnArrayChanged()
// with the appropriate args

Clients hook into the event and get called when things change. This is what most client code in .NET expects to have in an API ("call me when something happens"). They can hook into the code with something as simple as:

yourDownloader.ArrayChanged += (sender, e) =>
    Console.WriteLine(String.Format("Just downloaded {0} byte{1} at position {2}.",
            e.Length, e.Length == 1 ? "" : "s", e.Start));

Answer 5

.NET has a struct that does exactly what you want:

System.ArraySegment.

In any case, it's easy to implement it yourself too - just make a constructor that takes a base array, an offset, and a length. Then implement an indexer that offsets indexes behind the scenes, so your ArraySegment can be seamlessly used in the place of an array.

Answer 6

Whether this is needed or not depends on whether you can afford to accumulate all the data from a file before processing it, or whether you need to provide a streaming mode where you process each chunk as it arrives. This depends on two things: how much data there is (you probably would not want to accumulate a multi-gigabyte file), and how long it takes the file to completely arrive (if you are getting the data over a slow link you might not want your client to wait till it had all arrived). So it is a reasonable feature to add, depending on how the library is to be used. Streaming mode is usually a desirable attribute, so I would vote for implementing the feature. However, the idea of putting the data into an array seems wrong, because it fundamentally implies a non-streaming design, and because it requires an additional copy. What you could do instead is to keep each chunk of arriving data as a discrete piece. These could be stored in a container for which adding at the end and removing from the front is efficient.

Answer 7

I agree with the OP: sometimes you just plain need to pay some attention to efficiency. I don't think the example of providing an API is the best, because that certainly calls for leaning toward safety and simplicity over efficiency.

However, a simple example is when processing large numbers of huge binary files that have zillions of records in them, such as when writing a parser. Without using a mechanism such as System.ArraySegment, the parser becomes a big memory hog, and is greatly slowed down by creating a zillion new data elements, copying all the memory over, and fragmenting the heck out of the heap. It's a very real performance issue. I write these kinds of parsers all the time for telecommunications stuff which generate millions of records per day in each of several categories from each of many switches with variable length binary structures that need to be parsed into databases.

Using the System.ArraySegment mechanism versus creating new structure copies for each record tremendously speeds up the parsing, and greatly reduces the peak memory consumption of the parser. These are very real advantages because the servers run multiple parsers, run them frequently, and speed and memory conservation = very real cost savings in not having to have so many processors dedicated to the parsing.

System.Array segment is very easy to use. Here's a simple example of providing a base way to track the individual records in a typical big binary file full of records with a fixed length header and a variable length record size (obvious exception control deleted):

public struct MyRecord
{
    ArraySegment<byte> header;
    ArraySegment<byte> data;
}


public class Parser
{
    const int HEADER_SIZE = 10;
    const int HDR_OFS_REC_TYPE = 0;
    const int HDR_OFS_REC_LEN = 4;
    byte[] m_fileData;
    List<MyRecord> records = new List<MyRecord>();

    bool Parse(FileStream fs)
    {
        int fileLen = (int)fs.FileLength;
        m_fileData = new byte[fileLen];
        fs.Read(m_fileData, 0, fileLen);
        fs.Close();
        fs.Dispose();
        int offset = 0;
        while (offset + HEADER_SIZE < fileLen)
        {
            int recType = (int)m_fileData[offset];
            switch (recType) { /*puke if not a recognized type*/ }
            int varDataLen = ((int)m_fileData[offset + HDR_OFS_REC_LEN]) * 256
                     + (int)m_fileData[offset + HDR_OFS_REC_LEN + 1];
            if (offset + varDataLen > fileLen) { /*puke as file has odd bytes at end*/}
            MyRecord rec = new MyRecord();
            rec.header = new ArraySegment(m_fileData, offset, HEADER_SIZE);
            rec.data = new ArraySegment(m_fileData, offset + HEADER_SIZE,   
                          varDataLen);
            records.Add(rec);
            offset += HEADER_SIZE + varDataLen;
        } 
    }
}

The above example gives you a list with ArraySegments for each record in the file while leaving all the actual data in place in one big array per file. The only overhead are the two array segments in the MyRecord struct per record. When processing the records, you have the MyRecord.header.Array and MyRecord.data.Array properties which allow you to operate on the elements in each record as if they were their own byte[] copies.