Add 64 bit offset to a pointer

Question

In F#, there's the NativePtr module, but it seems to only support 32 bit offsets for its’ add/get/set functions, just like System.IntPtr does.

Is there a way to add a 64 bit offset to a native pointer (nativeptr<'a>) in F#? Of course I could convert all addresses to 64 bit integers, do normal integer operations and then convert the result again to nativeptr<'a>, but this would cost additional add and imul instructions. I really want the AGUs to perform the address calculations.

For instance, using unsafe in C# you could do something like

void* ptr = Marshal.AllocHGlobal(...).ToPointer();
int64 offset = ...;
T* newAddr = (T*)ptr + offset; // T has to be an unmanaged type

Well actually you can't, because there is no "unmanaged" constraint for type parameters, but at least you can do general pointer arithmetic in a non-generic way.

In F# we finally got the unmanaged constraint; but how do I do the pointer arithmetic?

Answer 1

Imaging data can easily exceed 4 GiB ;)

It can't, x64 code has a +/-2 GB offset restriction. One of the reasons you can't allocate arrays larger than 2GB in .NET. This restriction exists in unmanaged 64-bit C/C++ code as well. There are libraries that work around this restriction, like WIC, but addressing all of the bitmap bits directly doesn't make sense when you use them.

Nevertheless, it is possible to generate such an address by casting IntPtr to long in C#:

IntPtr addr = SomeCall();
long offset = blah;
addr = (IntPtr)((long)addr + offset);

Answer 2

I'm not an expert in this field, but I took a look at the F# implementation of the NativePtr module and I think that there is no performance overhead associated with converting nativeptr<'a> to nativeint and back.

The implementation uses inline IL and the inline IL code doesn't contain any code - it is there just to make the F# compiler think that the value on the stack has a different type:

let inline ofNativeInt (x:nativeint)    = (# "" x : nativeptr<_> #)
let inline toNativeInt (x:nativeptr<_>) = (# "" x : nativeint    #)

In fact, the NativePtr.add method also uses these two methods - it converts the pointer to nativeint and then adds the 32bit integer (multiplied by the size of the 'a type).

So, the following function should be fine:

let inline addNativeInt (x:nativeptr<'a>) (n:nativeint) : nativeptr<'a> = 
   (NativePtr.toNativeInt x) + n |> NativePtr.ofNativeInt

All functions used in the code should be inlined, so you'll end up with just a single instruction for addition (though, I have not verified that). You don't even have to worry about using the function multiple times in your code (you can work with nativeptr<'a> all the time and use this function for addition).

However, partitioning data may also be an option - as far as I know, the MSR team that was using F# for processing some large (>2GB) data sets used exactly this approach - they partitioned the data into 2GB blocks (stored in arrays).