Slow shared memory performance when moved to 64-bit OS

Question

I am having an issue with a 32-bit legacy app running on 64-bit windows. The app in question uses CreateFileMapping to create shared memory. When this is run on 64-bit Windows any attempt to access this shared memory from another process takes about 1 second. The shared memory is created using page protection flags:

flProtect = PAGE_READONLY | SEC_NOCACHE | SEC_COMMIT;

when the same memory is created using:

flProtect = PAGE_READONLY | SEC_COMMIT;

the issue disappears. For now this work around is acceptable, but we do have some devices that require the SEC_NOCACHE flag to be set.

Can someone enlighten me on why SEC_NOCACHE would affect performance in this situation?

Update: it seems that only writing to this buffer has increased to 1000ms. Reading does not seem to be affected. We are writing about 5MB to the buffer in this time.

Update2: This software is used on many systems, and one of the systems has a physical device that requires the use of this flags. We are currently limited to running the machine with this device in 32bit windows.

Answer 1

I would guess, that because the memory has to be remapped from 64-bit to 32-bit, it becomes expensive to provide a 'bounce' buffer. When caching is enabled, this bounce buffer is implicit and the OS may circumvent the need to continously update the memory section.

Answer 2

You are disabling the file system cache with that flag. Yes, that makes an enormous difference, it forces the OS to work with the disk driver and read sectors directly. Cylinders cannot be read and cached, disabling the optimization that makes reading tracks without having to move the read head so cheap. And lazy write-back is disabled, an optimization that makes disk writes appear instantaneous.

Answer 3

Here's what Microsoft has to say about that flag:

The SEC_NOCACHE flag is intended for architectures that require various locking structures to be located in memory that is not ever fetched into the CPU cache. On x86 and MIPS machines, use of this flag just slows down the performance because the hardware keeps the cache coherent.

Unfortunately they don't quantify the amount of slow down.