Marshalling a big-endian byte collection into a struct in order to pull out values

Question

There is an insightful question about reading a C/C++ data structure in C# from a byte array, but I cannot get the code to work for my collection of big-endian (network byte order) bytes. (EDIT: Note that my real struct has more than just one field.) Is there a way to marshal the bytes into a big-endian version of the structure and then pull out the values in the endianness of the framework (that of the host, which is usually little-endian)?

(Note, reversing the array of bytes will not work - each value's bytes must be reversed, which does not give you the same collection as reversing all of the bytes.)

This should summarize what I'm looking for (LE=LittleEndian, BE=BigEndian):

void Main()
{
    var leBytes = new byte[] {1, 0, 2, 0};
    var beBytes = new byte[] {0, 1, 0, 2};
    Foo fooLe = ByteArrayToStructure<Foo>(leBytes);
    Foo fooBe = ByteArrayToStructureBigEndian<Foo>(beBytes);
    Assert.AreEqual(fooLe, fooBe);
}

[StructLayout(LayoutKind.Explicit, Size=4)]
public struct Foo  {
    [FieldOffset(0)] 
    public ushort firstUshort;
    [FieldOffset(2)] 
    public ushort secondUshort;
}

T ByteArrayToStructure<T>(byte[] bytes) where T: struct 
{
    GCHandle handle = GCHandle.Alloc(bytes, GCHandleType.Pinned);
    T stuff = (T)Marshal.PtrToStructure(handle.AddrOfPinnedObject(),typeof(T));
    handle.Free();
    return stuff;
}

T ByteArrayToStructureBigEndian<T>(byte[] bytes) where T: struct 
{
    ???
}

Other helpful links:

Byte of a struct and onto endian concerns

A little more on bytes and endianness (byte order)

Read binary files more efficiently using C#

Unsafe and reading from files

Mono's contribution to the issue

Answer 1

Have you tried MiscUtil? It has a utility class named EndianBitConverter to convert between big and little endian byte arrays.

• http://www.yoda.arachsys.com/csharp/miscutil/

Answer 2

From my point of view you just need to add an Array.Reverse() before the conversion of the byte array.

Answer 3

I agree with @weismat and believe there is no solution.

What you show in your exemple is that you can access to a raw byte buffer as if it where any OTHER structure without changing anything to it, not copying or moving data around, nothing. Just pinning it to avoid it to move around because of GC.

This is basically what you usually achive in C by using a union type containing both your target structure and a byte array of the same size.

The good side is that it is really efficient.

That has several drawbacks, the main one being that you can only access this way to data that are in the native machine order (be it LE or BE). Hence your ByteArrayToStructure is not really LE, it is only so because your underneath processor is LE. If you compile the same program on another target that happen to be BE it works go the other way and believe your byte array is BE. Other drawbacks are that you must be very cautious with data alignment, be aware of possible padding, etc. and of course that there is no way to change byte order from LE to BE without moving data in bytes array (if you have a 16 bits integers only array as in your example this is merely swapping every two bytes).

I happened to have a similar problem and poundered not to use this solution because of the previous drawbacks and opted to hide my input structures behind accessors to hide access to underneath bytes array. It may not be as elegant but it is simple and also avoid to copy buffer of move data in any way.

Answer 4

It seems there must be a more elegant solution, but this should at least get you going:

    static T ByteArrayToStructureBigEndian<T>(byte[] bytes) where T : struct
    {
        GCHandle handle = GCHandle.Alloc(bytes, GCHandleType.Pinned);
        T stuff = (T)Marshal.PtrToStructure(handle.AddrOfPinnedObject(), typeof(T));
        handle.Free();
        System.Type t = stuff.GetType();
        FieldInfo[] fieldInfo = t.GetFields();
        foreach (FieldInfo fi in fieldInfo)
        {                 
            if (fi.FieldType == typeof(System.Int16))
            {
                // TODO
            }
            else if (fi.FieldType == typeof(System.Int32))
            {
                // TODO
            }
            else if (fi.FieldType == typeof(System.Int64))
            {
                // TODO
            }
            else if (fi.FieldType == typeof(System.UInt16))
            {
                UInt16 i16 = (UInt16)fi.GetValue(stuff);
                byte[] b16 = BitConverter.GetBytes(i16);
                byte[] b16r = b16.Reverse().ToArray();
                fi.SetValueDirect(__makeref(stuff), BitConverter.ToUInt16(b16r, 0);
            }
            else if (fi.FieldType == typeof(System.UInt32))
            {
                // TODO
            }
            else if (fi.FieldType == typeof(System.UInt64))
            {
                // TODO
            }
        }
        return stuff;
    }

Answer 5

The traditional solution is to use ntohl() and ntohs().

typedef struct {
  long foo;
  short bar, baz;
  char xyzzy;
} Data;

Data d;
memcpy(&d, buffer, sizeof(Data));

d.foo = ntohl(d.foo);
d.bar = ntohs(d.bar);
d.baz = ntohs(d.baz);
// don't need to change d.xyxxy

The above works on any platform that has BSD Sockets, no matter whether it's big-endian, little-endian, or something utterly weird like a VAX. The reverse operation is done using hton*().

On big-endian platforms the functions are usually no-ops and should therefore be zero cost.

Answer 6

I finally figured out a way that didn't involve reflection and is mostly user-friendly. It uses Mono's DataConverter class (source) which, unfortunately, is fairly buggy at this point. (For example, floats and doubles don't seem to work correctly, string parsing is broken, etc.)

The trick is to Unpack and re-pack the bytes as big-endian, which requires a string describing what types are in the byte array (see the last method).Also, byte alignment is tricky: there are four bytes in the struct instead of one because marshaling seems to rely on having 4-byte alignment (I still don't quite understand that part). (EDIT: I have found that adding Pack=1 to the StructLayout attribute usually takes care of byte-alignment issues.)

Note, this sample code was used in LINQPad - the Dump extension method just prints info about the object and returns the object (it's fluent).

public void Main()
{
    var beBytes = new byte[] {
        0x80, 
        0x80, 
        0x80, 
        0x80, 
        0x80,0, 
        0x80,0, 
        0x80,0,0,0, 
        0x80,0,0,0,
        0x80,0,0,0,0,0,0,0, 
        0x80,0,0,0,0,0,0,0, 
//      0,0,0x80,0x3F, // float of 1
//      0,0,0,0,0,0,0xF0,0x3F, // double of 1
        0x54,0x65,0x73,0x74,0x69,0x6E,0x67,0,0,0 // Testing\0\0\0
    };
    var leBytes = new byte[] {
        0x80, 
        0x80, 
        0x80, 
        0x80, 
        0,0x80,
        0,0x80, 
        0,0,0,0x80,
        0,0,0,0x80, 
        0,0,0,0,0,0,0,0x80, 
        0,0,0,0,0,0,0,0x80, 
//      0,0,0x80,0x3F, // float of 1
//      0,0,0,0,0,0,0xF0,0x3F, // double of 1
        0x54,0x65,0x73,0x74,0x69,0x6E,0x67,0,0,0 // Testing\0\0\0
    };
    Foo fooLe = ByteArrayToStructure<Foo>(leBytes).Dump("LE");
    Foo fooBe = ByteArrayToStructureBigEndian<Foo>(beBytes, 
        "bbbbsSiIlL"
//      + "fd" // float, then double
        +"9bb").Dump("BE");
    Assert.AreEqual(fooLe, fooBe);
}

[StructLayout(LayoutKind.Sequential, Pack = 1)]
public struct Foo  {
    public byte b1;
    public byte b2;
    public byte b3;
    public byte b4;
    public short s;
    public ushort S;
    public int i;
    public uint I;
    public long l;
    public ulong L;
//  public float f;
//  public double d;
    [MarshalAs(UnmanagedType.ByValTStr, SizeConst = 10)]
    public string MyString;
}

T ByteArrayToStructure<T>(byte[] bytes) where T: struct 
{
    GCHandle handle = GCHandle.Alloc(bytes, GCHandleType.Pinned);
    T stuff = (T)Marshal.PtrToStructure(handle.AddrOfPinnedObject(),typeof(T));
    handle.Free();
    return stuff;
}

T ByteArrayToStructureBigEndian<T>(byte[] bytes, string description) where T: struct 
{
    byte[] buffer = bytes;
    IList unpacked = DataConverter.Unpack("^"+description, buffer, 0).Dump("unpacked");
    buffer = DataConverter.PackEnumerable("!"+description, unpacked).Dump("packed");
    return ByteArrayToStructure<T>(buffer);
}