How to alloc more than MaxInteger bytes of memory c#

Question

Hi,

I wish to allocate more than MaxInteger bytes of memory.

Marshall.AllocHGlobal() expects an integer - so I cannot use this. Does anyone know of another way?

Answer 1

This is not possible from managed code without a pinvoke call and for good reason. Allocating that much memory is usually a sign of a bad solution that needs revisiting.

Can you tell us why you think you need this much memory?

Answer 2

That's not possible on current mainstream hardware. Memory buffers are restricted to 2 gigabytes, even on 64-bit machines. Indexed addressing of the buffer is still done with a 32-bit signed offset. It is technically possible to generate machine code that can index more, using a register to store the offset, but that's expensive and slows down all array indexing, even for the ones that aren't larger than 2 GB.

Furthermore, you can't get a buffer larger than about 650MB out of the address space available to a 32-bit process. There aren't enough contiguous memory pages available because virtual memory contains both code and data at various addresses.

Companies like IBM and Sun sell hardware that can do this.

Answer 3

Use Marshal.AllocHGlobal(IntPtr). This overload treats the value of the IntPtr as the amount of memory to allocate and IntPtr can hold a 64 bit value.

Answer 4

I don't know the answer, but I would suggest that you should try another way to do what you are trying to achieve. IMHO, You should consider implementing some kind of data structure to handle your data instead of allocating huge buffer. Of course it is more complicated to do so, but sometimes it is inevitable when you are trying to do non-trivial data processing.

Answer 5

Hi Chris,

I really appreciate the effort you put into this answer.

I changed the platform to x64 then I ran the code below.

myp appears to have the right length: about 3.0G. But stubbornly "buffer" maxes out at 2.1G

Any idea why?

      var fileStream = new FileStream("C:\\big.BC2",
          FileMode.Open,
          FileAccess.Read,
          FileShare.Read,
          16 * 1024,
          FileOptions.SequentialScan);
        Int64 length = fileStream.Length;
        Console.WriteLine(length);
        Console.WriteLine(Int64.MaxValue);
        IntPtr myp = new IntPtr(length);
        //IntPtr buffer = Marshal.AllocHGlobal(myp);
        IntPtr buffer = VirtualAllocEx(
            Process.GetCurrentProcess().Handle,
            IntPtr.Zero,
            new IntPtr(length),
            AllocationType.Commit | AllocationType.Reserve,
            MemoryProtection.ReadWrite); 
        unsafe
        {
            byte* pBytes = (byte*)myp.ToPointer(); 
            var memoryStream = new UnmanagedMemoryStream(pBytes, (long)length, (long)length, FileAccess.ReadWrite);
            fileStream.CopyTo(memoryStream);

Answer 6

From a comment:

How do I create a second binaryreader that can read the same memorystream independantly?

var fileStream = new FileStream("C:\\big.BC2",
      FileMode.Open,
      FileAccess.Read,
      FileShare.Read,
      16 * 1024,
      FileOptions.SequentialScan);
    Int64 length = fileStream.Length;
    IntPtr buffer = Marshal.AllocHGlobal(length);
    unsafe
    {
        byte* pBytes = (byte*)myp.ToPointer(); 
        var memoryStream = new UnmanagedMemoryStream(pBytes, (long)length, (long)length, FileAccess.ReadWrite);
        var binaryReader = new BinaryReader(memoryStream);
        fileStream.CopyTo(memoryStream);
        memoryStream.Seek(0, SeekOrigin.Begin);
        // Create a second UnmanagedMemoryStream on the _same_ memory buffer
        var memoryStream2 = new UnmanagedMemoryStream(pBytes, (long)length, (long)length, FileAccess.Read);
        var binaryReader2 = new BinaryReader(memoryStream);
     }

Answer 7

If you can't make it work the way you want it to directly, create a class to provide the type of behaviour you want. So, to use big arrays:

using System;
using System.Collections.Generic;
using System.Text;
using System.IO;

namespace BigBuffer
{
  class Storage
  {
    public Storage (string filename)
    {
      m_buffers = new SortedDictionary<int, byte []> ();
      m_file = new FileStream (filename, FileMode.Open, FileAccess.Read, FileShare.Read);
    }

    public byte [] GetBuffer (long address)
    {
      int
        key = GetPageIndex (address);

      byte []
        buffer;

      if (!m_buffers.TryGetValue (key, out buffer))
      {
        System.Diagnostics.Trace.WriteLine ("Allocating a new array at " + key);
        buffer = new byte [1 << 24];
        m_buffers [key] = buffer;

        m_file.Seek (address, SeekOrigin.Begin);
        m_file.Read (buffer, 0, buffer.Length);
      }

      return buffer;
    }

    public void FillBuffer (byte [] destination_buffer, int offset, int count, long position)
    {
      do
      {
        byte []
          source_buffer = GetBuffer (position);

        int
          start = GetPageOffset (position),
          length = Math.Min (count, (1 << 24) - start);

        Array.Copy (source_buffer, start, destination_buffer, offset, length);

        position += length;
        offset += length;
        count -= length;
      } while (count > 0);
    }

    public int GetPageIndex (long address)
    {
      return (int) (address >> 24);
    }

    public int GetPageOffset (long address)
    {
      return (int) (address & ((1 << 24) - 1));
    }

    public long Length
    {
      get { return m_file.Length; }
    }

    public int PageSize
    {
      get { return 1 << 24; }
    }

    FileStream
      m_file;

    SortedDictionary<int, byte []>
      m_buffers;
  }

  class BigStream : Stream
  {
    public BigStream (Storage source)
    {
      m_source = source;
      m_position = 0;
    }

    public override bool CanRead
    {
      get { return true; }
    }

    public override bool CanSeek
    {
      get { return true; }
    }

    public override bool CanTimeout
    {
      get { return false; }
    }

    public override bool CanWrite
    {
      get { return false; }
    }

    public override long Length
    {
      get { return m_source.Length; }
    }

    public override long Position
    {
      get { return m_position; }
      set { m_position = value; }
    }

    public override void Flush ()
    {
    }

    public override long Seek (long offset, SeekOrigin origin)
    {
      switch (origin)
      {
      case SeekOrigin.Begin:
        m_position = offset;
        break;

      case SeekOrigin.Current:
        m_position += offset;
        break;

      case SeekOrigin.End:
        m_position = Length + offset;
        break;
      }

      return m_position;
    }

    public override void SetLength (long value)
    {
    }

    public override int Read (byte [] buffer, int offset, int count)
    {
      int
        bytes_read = (int) (m_position + count > Length ? Length - m_position : count);

      m_source.FillBuffer (buffer, offset, bytes_read, m_position);

      m_position += bytes_read;
      return bytes_read;
    }

    public override void  Write(byte[] buffer, int offset, int count)
    {
    }

    Storage
      m_source;

    long
      m_position;
  }

  class IntBigArray
  {
    public IntBigArray (Storage storage)
    {
      m_storage = storage;
      m_current_page = -1;
    }

    public int this [long index]
    {
      get
      {
        int
          value = 0;

        index <<= 2;

        for (int offset = 0 ; offset < 32 ; offset += 8, ++index)
        {
          int
            page = m_storage.GetPageIndex (index);

          if (page != m_current_page)
          {
            m_current_page = page;
            m_array = m_storage.GetBuffer (m_current_page);
          }

          value |= (int) m_array [m_storage.GetPageOffset (index)] << offset;
        }

        return value;
      }
    }

    Storage
      m_storage;

    int
      m_current_page;

    byte []
      m_array;
  }

  class Program
  {
    static void Main (string [] args)
    {
      Storage
        storage = new Storage (@"<some file>");

      BigStream
        stream = new BigStream (storage);

      StreamReader
        reader = new StreamReader (stream);

      string
        line = reader.ReadLine ();

      IntBigArray
        array = new IntBigArray (storage);

      int
        value = array [0];

      BinaryReader
        binary = new BinaryReader (stream);

      binary.BaseStream.Seek (0, SeekOrigin.Begin);

      int
        another_value = binary.ReadInt32 ();
    }
  }
}

I split the problem into three classes:

• Storage - where the actual data is stored, uses a paged system
• BigStream - a stream class that uses the Storage class for its data source
• IntBigArray - a wrapper around the Storage type that provides an int array interface

The above can be improved significantly but it should give you ideas about how to solve your problems.