Having trouble with extension methods for byte arrays

Question

I'm working with a device that sends back an image, and when I request an image, there is some undocumented information that comes before the image data. I was only able to realize this by looking through the binary data and identifying the image header information inside.

I originally had a normal method and converted it to an extension method. The original question here was related to the compiler complaining about not having Array as the first parameter (I had Byte[]), but it turns out that I had made an error and forgot to delete the first argument in the calling code. In other words, I used to have:

Byte[] new_buffer = RemoveUpToByteArray(buffer, new byte[] { 0x42, 0x4D });

and after changing to an extension method, I had erroneously used:

buffer.RemoveUpToByteArray( buffer, new byte[] { 0x42, 0x4D });

Anyhow, that's all fixed now because I realized my mistake as I was entering the code example into SO. However, I have a new problem that is simply lack of understanding of extension methods and reference vs. value types. Here's the code:

public static void RemoveFromByteArrayUntil(this Byte[] array, Byte[] until)
{
    Debug.Assert(until.Count() > 0);
    int num_header_bytes = until.Count();
    int header_start_pos = 0; // the position of the header bytes, defined by [until]
    byte first_header_byte = until[0];
    while(header_start_pos != -1) {
        header_start_pos = Array.IndexOf(array, first_header_byte, header_start_pos);
        if(header_start_pos == -1)
            break;
        // if we get here, then we've found the first header byte, and we need to look
        // for the next ones sequentially
        for(int header_ctr=1; header_ctr<num_header_bytes; header_ctr++) {
            // we're going to loop over each of the header bytes, but will
            // bail out of this loop if there isn't a match
            if(array[header_start_pos + header_ctr] != until[header_ctr]) {
                // no match, so bail out.  but before doing that, advance
                // header_start_pos so the outer loop won't find the same
                // occurrence of the first header byte over and over again
                header_start_pos++;
                break;
            }
        }
        // if we get here, we've found the header!
        // create a new byte array of the new size
        int new_size = array.Count() - header_start_pos;
        byte[] output_array = new byte[new_size];
        Array.Copy(array, header_start_pos, output_array, 0, new_size);
        // here is my problem -- I want to change what array points to, but
        // when this code returns, array goes back to its original value, which
        // leads me to believe that the first argument is passed by value.
        array = output_array;
        return;
    }
    // if we get here, we didn't find a header, so throw an exception
    throw new HeaderNotInByteArrayException();
}

My problem now is that it looks like the first this argument to the extension method is passed by value. I want to reassign what array points to, but in this case, it looks like I'll have to just manipulate array's data instead.

Answer 1

Extension methods are static methods that only appear to be instance methods. You can consider the instance the extension method is working on to be read only (by value). Assigning to the instance method of byte[] that is the first parameter of your extension won't work. You won't be able to get away from assigning, but you could modify your extension then write your assignment like this:

buffer = buffer.RemoveUpToByteArray(header);

Make your extension return the byte array result, and don't try to assign to buffer within the extension. Your extension would then be something like this:

public static class MyExtensionMethods
{
    public static byte[] RemoveUpToByteArray(this byte[] buffer, byte[] header)
    {
        byte[] result = buffer;

        // your logic to remove header from result

        return result;
    }
}

I hope this helps.

EDIT: The above is correct for value types only. If the type you are extending is a reference type, then you would not have an issue operating directly on the type like you are trying to do above. Sadly, a byte array is a struct, and thus derived from System.ValueType. Consider the following, which would be perfectly legal inside an extension, and would give the desired result:

public class MyBytes
{
    public byte[] ByteArray { get; set; }
}

public static class MyExtensionMethods
{
    // Notice the void return here...
    public static void MyClassExtension(this MyBytes buffer, byte[] header)
    {
        buffer.ByteArray = header;
    }
}

Answer 2

Hi Dave, I am sorry that I do not know what specific problem you are encountering, or how to resolve it [certainly checking namespace is referenced and resolving any conflicts with similarly named methods is a start], but I did notice one or two oddities.

Consider the following sample solution,

using System.Linq;
namespace Sample.Extensions
{
    public static class ByteExtensions
    {
        public static void RemoveHeader (this byte[] buffer, byte[] header)
        {
            // take first sequence of bytes, compare to header, if header
            // is present, return only content
            // 
            // NOTE: Take, SequenceEqual, and Skip are standard Linq extensions
            if (buffer.Take (header.Length).SequenceEqual (header))
            {
                buffer = buffer.Skip (header.Length).ToArray ();
            }
        }
    }
}

This compiles and runs in VS2010RC. To demonstrate usage,

using Sample.Extensions;
namespace Sample
{
    class Program
    {
        static void Main (string[] args)
        {
            byte[] buffer = new byte[] { 00, 01, 02 };
            byte[] header = new byte[] { 00, 01 };
            buffer.RemoveHeader (header);

            // hm, so everything compiles and runs, but buffer == { 00, 01, 02 }
        }
    }
}

So we will not receive a compile or run-time error but clearly it will not operate as intended. This is because extensions must still comply with standard method semantics, meaning parameters are passed by value. We cannot change buffer to point to our new array.

We can resolve this issue by rewriting our method to conventional function semantics,

public static byte[] RemoveHeaderFunction (this byte[] buffer, byte[] header)
{
    byte[] stripped = null;
    if (stripped.Take (header.Length).SequenceEqual (header))
    {
        stripped = stripped.Skip (header.Length).ToArray ();
    }
    else
    {
        stripped = buffer.ToArray ();
    }
    return stripped;
}

Now

using Sample.Extensions;
namespace Sample
{
    class Program
    {
        static void Main (string[] args)
        {
            byte[] buffer = new byte[] { 00, 01, 02 };
            byte[] header = new byte[] { 00, 01 };

            // old way, buffer will still contain { 00, 01, 02 }
            buffer.RemoveHeader (header);

            // new way! as a function, we obtain new array of { 02 }
            byte[] stripped = buffer.RemoveHeaderFunction (header);
        }
    }
}

Unfortunately, arrays are immutable value types [may be using these terms incorrectly]. The only way to modify your "array" in-place is to change the container to a mutable reference-type, like a List<byte>.

If you are really keen on "passing by ref", in-place, side-effect semantics, then one option may be the following

using System.Linq;
namespace Sample.Extensions
{
    public static class ListExtensions
    {
        public static void RemoveHeader<T> (this List<T> list, List<T> header)
        {
            if (list.Take (header.Count).SequenceEqual (header))
            {
                list.RemoveRange (0, header.Count);
            }
        }
    }
}

As for usage,

static void Main (string[] args)
{
    byte[] buffer = new byte[] { 00, 01, 02 };
    byte[] header = new byte[] { 00, 01 };

    List<byte> bufferList = buffer.ToList ();

    // in-place side-effect header removal
    bufferList.RemoveHeader (header.ToList ());
}

Under the hood, List<T> is maintaining an array of type T. At certain thresholds, it is simply manipulating the underlying array and\or instantiating new arrays for us.

Hope this helps! :)