Operations and functions that increase Virtual Bytes

Question

Having some out-of-memory problems with a 32-bit process in Windows I begun using Performance Monitor to log certain counters for that process.

Though it is normal that Virtual Bytes is higher than both Private Bytes and Working Set, I found that in my case there was a substantial difference, Virtual Bytes was much higher than both Private Bytes and Working Set.

What specific operations and Win32/CRT functions (in C or C++) would increase Virtual Bytes but not Private Bytes and Working Set?

I guess it would be some kind of shared resources, if I understand the description of the different counters in Performance Monitor.

---

As there seems to be some (to say the least) confusion on the naming convention to use for the memory counters in different releases of Windows as well in different applications in the same release of Windows, I put together the following:

Information from MSDN

According to MSDN - Memory Limits for Windows Releases, the user-mode virtual address space limit in 32-bit Windows for each 32-bit process is normally 2 GB. It can be up to 3 GB with IMAGE_FILE_LARGE_ADDRESS_AWARE and 4GT.

Below is a description of the different counters in Performance Monitor along with the corresponding columns in Task Manager and the Win32 structure which holds the information, according to MSDN - Memory Performance Information.

Virtual Bytes

Virtual Bytes is the current size, in bytes, of the virtual address space the process is using. Use of virtual address space does not necessarily imply corresponding use of either disk or main memory pages. Virtual space is finite, and the process can limit its ability to load libraries.

Task Manager XP: N/A
Task Manager Vista: N/A
Structure: MEMORYSTATUSEX.ullTotalVirtual-MEMORYSTATUSEX.ullAvailVirtual

Private Bytes

Private Bytes is the current size, in bytes, of memory that this process has allocated that cannot be shared with other processes.

Task Manager XP: VM Size
Task Manager Vista: Commit Size
Structure: PROCESS_MEMORY_COUNTERS_EX.PrivateUsage

Working Set

Working Set is the current size, in bytes, of the Working Set of this process. The Working Set is the set of memory pages touched recently by the threads in the process. If free memory in the computer is above a threshold, pages are left in the Working Set of a process even if they are not in use. When free memory falls below a threshold, pages are trimmed from Working Sets. If they are needed they will then be soft-faulted back into the Working Set before leaving main memory.

Task Manager XP: Mem Usage
Task Manager Vista: Working Set
Structure: PROCESS_MEMORY_COUNTERS_EX.WorkingSetSize

Answer 1

What is your programming language?

In managed frameworks, private bytes represent data that is allocated by unmanaged resources. Whereas virtual bytes represent the total memory usage (unmanaged and managed data).

Thus, it is very common to see substantial differences between private and virtual bytes in such frameworks.

Answer 2

By using VirtualAlloc, you can allocate virtual address space without actually allocating any physical memory. That should increase "virtual byte" count, but not your working set size.

The out of memory could be caused by running of address space because reserving too much address space.

Answer 3

Things that (may) increase virtual bytes without increasing private bytes I can think of right now:

• Binaries are often shared (i.e. not private), but occupy significant address space. This can be even larger than the size of the binary

• Using VirtualAlloc to reserve sequential address space without committing / accessing it. Custom Memory Managers might do that.

• Using a memory mapped file (without completely accessing it)