"System.TypeLoadException: Internal limitation: too many fields." With Large Static Data Structure

Question

I have a small C# library with a static class that holds a very large, but simple four-dimensional byte array representing a multi-dimensional decision table (about 90K bytes in total).

There's one other data structure of note in there, a Dictionary which helps index into the first dimension of the decision table.

This decision table is a static private data member, initialized by array static initializers. The code is generated in Visual Studio 2010 with T4, from Excel documents.

Decisions are obtained via a static method that indexes into the multi-dimensional array.

When I run a simple test app using this library, it bombs with a "System.TypeLoadException: Internal limitation: too many fields." Exception, at the first invocation of the static decision method.

The one remotely related topic on Stackoverflow mentions libraries with "too many symbols". I may be wrong but my library seems to have very few symbols indeed.

What's going on here?

Code Snippet:

        private static byte[][][][] decisions = new byte[][][][] {
    new byte[][][] { 
    new byte[][] { 
    new byte[]{5,6,6},new byte[]{5,6,6},
    new byte[]{5,6,6},new byte[]{5,6,6},new byte[]{5,6,6},
    new byte[]{5,6,6},new byte[]{5,6,6},new byte[]{5,6,6},new byte[]{5,6,6},
    new byte[]{5,6,6},new byte[]{5,6,6},new byte[]{5,6,6},new byte[]{5,6,6},new byte[]{5,6,6},
    new byte[]{5,6,6},new byte[]{5,6,6},new byte[]{5,6,6},new byte[]{5,6,6},new byte[]{5,6,6},new byte[]{5,6,6},
    new byte[]{5,6,6},new byte[]{5,6,6},new byte[]{5,6,6},new byte[]{5,6,6},new byte[]{5,6,6},new byte[]{5,6,6},new byte[]{5,6,6},
    new byte[]{5,6,6},new byte[]{5,6,6},new byte[]{5,6,6},new byte[]{5,6,6},new byte[]{5,6,6},new byte[]{5,6,6},new byte[]{5,6,6},new byte[]{5,6,6},
    new byte[]{5,6,6},new byte[]{5,6,6},new byte[]{5,6,6},new byte[]{5,6,6},new byte[]{5,6,6},new byte[]{5,6,6},new byte[]{5,6,6},new byte[]{5,6,6},new byte[]{5,6,6}

    },
    new byte[][] { 
    new byte[]{5,6,6},new byte[]{5,6,6},
    new byte[]{5,6,6},new byte[]{5,6,6},new byte[]{5,6,6},
    new byte[]{5,6,6},new byte[]{5,6,6},new byte[]{5,6,6},new byte[]{5,6,6},
    new byte[]{5,6,6},new byte[]{5,6,6},new byte[]{5,6,6},new byte[]{5,6,6},new byte[]{5,6,6},
    new byte[]{5,6,6},new byte[]{5,6,6},new byte[]{5,6,6},new byte[]{5,6,6},new byte[]{5,6,6},new byte[]{5,6,6},
    new byte[]{5,6,6},new byte[]{5,6,6},new byte[]{5,6,6},new byte[]{5,6,6},new byte[]{5,6,6},new byte[]{5,6,6},new byte[]{5,6,6},
    new byte[]{5,6,6},new byte[]{5,6,6},new byte[]{5,6,6},new byte[]{5,6,6},new byte[]{5,6,6},new byte[]{5,6,6},new byte[]{5,6,6},new byte[]{5,6,6},
    new byte[]{5,6,6},new byte[]{5,6,6},new byte[]{5,6,6},new byte[]{5,6,6},new byte[]{5,6,6},new byte[]{5,6,6},new byte[]{5,6,6},new byte[]{5,6,6},new byte[]{5,6,6}

    },

...and so on...

Answer 1

Wow. Looks fun. I can't comment on that specific error (although I expect it is actually too many intermediate locals), but in something remarkably similar (codegen based decision engine) I populated the data via serialisation (binary in my case, but any should work). This meant a two step load (create obj, load from file) but it worked very well.

As an aside: My constructor was enough to make reflector explode in a messy pile (very terminal error) so I feel your pain.

Answer 2

new byte[]{5,6,6}

The problem here is that the initializer {5, 6, 6} creates a static field. You can see it with Ildasm.exe. The CLR imposes a maximum of, I believe, 65535 fields in a class. Your auto-generated code is exceeding it.

You are going to have to do this differently. A file jumps to mind.

Answer 3

You could try emitting the code in the static constructor explicitly and index directly in the sub-arrays rather than using array initializer syntax. This will prevent the creation of the static fields and instead encode all of the data setup into a massive blob of IL. I'm not sure if there are similar limits on the amount of IL that can go in a single method, but I'm sure you'll find out if you modify the T4 accordingly.

It would look like:

static MyClass()
{
    decisions = new byte[N1][][][];
    decisions[0] = new byte[N2][][];
    ....
    decisions[0][0][0][0] = 5;
    decisions[0][0][0][1] = 6;
    decisions[0][0][0][2] = 6;
    ...
}