Options for Big Arrays in .NET with over 2^31 keys.

Question

Are there any built it options for arrays supporting more than 2^31 keys. This would make keys of type System.Int64 rather than System.Int. While 2^31 is a fair amount, there are a growing number of of applications where more keys would help. For instance, there are over 3 billion base pairs in the human genome. Any easy options?

Answer 1

It's not possible in .NET because there is a maximum limit of 2GB per object. Even in 64-bit, you still have this limit.

If you really want to try to keep all this data in memory at once, you could partition your data into chunks. You can even write a wrapper class to transparently map the index from an Int64 to an Int32 index on the correct partition.

You could read this article about BigArray.

Answer 2

Use unsafe code

Works only for basic types like int, double, ulong etc.!

Manually allocate memory using AllocHGlobal from System.Runtime.InteropServices.Marshal and use the familiar array syntax to access individual elements.

var rand = new Random();
const ulong n = (3UL*Giga/sizeof(double));

Console.WriteLine("Allocate memory for {0} doubles.",n);

//Need to cast number of bytes to IntPtr because int is too small
var p = (double*) Marshal.AllocHGlobal((IntPtr)(n*sizeof(double)));

Console.WriteLine("Fill memory");
for (var i = 0UL; i < n; i++)
    p[i] = rand.NextDouble();

Console.WriteLine("Take sum");
var s = 0.0;
for (var i = 0UL; i < n; i++)
    s += p[i];
Console.WriteLine("The total is {0}.",s);

Marshal.FreeHGlobal((IntPtr) p); //<--- Important! ^^

Of course that p-Pointer is not an actual array. You can't use it where arrays or IEnumerable<double>s are expected.

You could wrap this in a neat class - say UnsafeArray - that implements the IDisposable-pattern and provides a ulong index property.

Answer 3

We have similar object sizes. We created a compressed array which stores data in the way it takes minimum RAM. I think it is easy to create special storage for simple types like integer. Your users will thank you thousands of times.