Are there compiler flags to get malloc to return pointers above the 4G limit for 64bit testing (various platforms)?

Question

I need to test code ported from 32bit to 64bit where pointers are cast around as integer handles, and I have to make sure that the correct sized types are used on 64 bit platforms.

Are there any flags for various compilers, or even flags at runtime which will ensure that malloc returns pointer values greater than the 32bit limit?

Platforms I'm interested in:

1. Visual Studio 2008 on Windows XP 64, and other 64 bit windows
2. AIX using xLC
3. 64bit gcc
4. 64bit HP/UX using aCC

---

Sample Application that allocates 4GB

So thanks to R Samuel Klatchko's answer, I was able to implement a simple test app that will attempt to allocate pages in the first 4GB of address space. Hopefully this is useful to others, and other SO users can give me an idea how portable/effective it is.

#include <stdlib.h>
#include <stdio.h>
#define UINT_32_MAX 0xFFFFFFFF

#ifdef WIN32
typedef unsigned __int64 Tuint64;
#include <windows.h>
#else
typedef unsigned long long Tuint64;
#include <sys/mman.h>
#include <unistd.h>
#include <errno.h>
#include <string.h>
#endif

static void* Allocate(void* pSuggested, unsigned int PageSize)
{
#ifdef WIN32
   void* pAllocated = ::VirtualAlloc(pSuggested, PageSize, MEM_RESERVE ,PAGE_NOACCESS);
   if (pAllocated)
   {
      return pAllocated;
   }
   return (void*)-1;
#else
   void* pAllocated = ::mmap(pSuggested,
                             PageSize,
                             PROT_NONE,
                             MAP_PRIVATE | MAP_ANONYMOUS | MAP_NORESERVE, 
                             -1, 
                             0);
   if (pAllocated == MAP_FAILED)
   {
      pAllocated = (void*)-1;
   }
   return pAllocated;
#endif
}

static void Deallocate(void* pRegion, unsigned int PageSize)
{
#ifdef WIN32
   ::VirtualFree(pRegion,0,MEM_RELEASE);
#else
   ::munmap(pRegion,PageSize);
#endif
}

static void Gobble32bitAddressSpace()
{
#ifdef WIN32
   SYSTEM_INFO SysInfo;
   ::GetSystemInfo(&SysInfo);
   unsigned int PageSize = SysInfo.dwAllocationGranularity;
#else
   unsigned int PageSize = ::sysconf(_SC_PAGE_SIZE);
#endif

   unsigned int AllocatedPages = 0;
   unsigned int SkippedPages = 0;
   void *pStart = 0;
   while( ((Tuint64)pStart) < UINT_32_MAX)
   {
      void* pAllocated = Allocate(pStart, PageSize);
      if (pAllocated  != (void*)-1)
      {
         if (pAllocated == pStart)
         {
            //Allocated at expected location
            AllocatedPages++;
         }
         else 
         {
            //Allocated at a different location
            //unallocate and consider this page unreserved
            SkippedPages++;
            Deallocate(pAllocated,PageSize);
         }
      }
      else
      {
         //could not allocate at all
         SkippedPages++;
      }
      pStart = (char*)pStart + PageSize;
   }
   printf("PageSize : %u\n",PageSize);
   printf("Allocated Pages : %u (%u bytes)\n",AllocatedPages,PageSize*AllocatedPages);
   printf("Skipped Pages : %u (%u bytes)\n",SkippedPages,SkippedPages*PageSize);
}

int main()
{
   Gobble32bitAddressSpace();

   //Try to call malloc now and see if we get an
   //address above 4GB
   void* pFirstMalloc = ::malloc(1024);
   if (((Tuint64)pFirstMalloc) >= UINT_32_MAX)
   {
      printf("OK\n");
   }
   else
   {
      printf("FAIL\n");
   }
   return 0;
}

Answer 1

One technique I have used in the past is to allocate enough memory at startup that all the address space below the 4GB limit is used up. While this technique does rely on malloc first using the lower parts of the address space, this was true on all the platforms I work on (Linux, Solaris and Windows).

Because of how Linux uses overcommit, if you don't touch the allocated space below the 4GB limit, you won't use up any virtual memory.

On Windows, you can use VirtualAlloc() with the MEM_RESERVE flag to use up address space without allocating any actual storage.

Answer 2

You would do well to rewrite your code so that the intptr_t type were used, since that is intended exactly to render such practices safer. Unfortunately it is defined in the c99 header, and VC++ does not support C99. That would not however stop you from creating such a header for that platform.

You might also add asserts where such casts occur e.g.

assert( sizeof(integer_param) == sizeof(void*) ) ;

or you could cast the value back to the original pointer type, and then compare:

assert( (mytype*)integer_param == original_pointer ) ;

Answer 3

Not a compiler switch, but a boot-time switch for Windows can do what you want. There is a command called "nolomem" which forces everything to be loaded in address space > 4GB.

If you are using XP, you should be able to use /nolomem in boot.ini . See documentation on OSR.

For Vista/Win7 you can use NOLOMEM option. Documentation is here. This can be done as such:

bcdedit /set {current} NOLOMEM

Answer 4

Not that you asked specifically, but for others that might be curious, gcc on Mac OS X seems to allocate from the area above 4GB for 64-bit programs by default.

Here's a C program to verify this on whatever compiler/OS combination you might have:

#include <stdlib.h>
#include <stdio.h>

int main() {
    void *p = malloc(1000);
    printf("%p\n", p);
    return 0;
}