Use Process Explorer instead of Task Manager to look at the memory consumption of your application.
- Private Bytes: is the amount of memory that is used by your application. This is also the value shown by Task Manager in the VM Size column (under XP)
- Virtual Memory Size: this is the largest address that is currently in use by your application. This value is limited by 2 GB (for a 32-bit application in a 32-bit OS).
The different between Private Bytes and Virtual Memory Size is caused by memory fragmentation. The only thing you can do about it is to use a memory manager that tries to reduce fragmentation, e.g. by using memory pools.
The DougLea memory manager is a free alternative. There are also commercial memory manager. You can also try to write your own memory manager (it's not that hard: you only have to override the new and delete operators).
There are also tricks to increase the limit of 2 GB. If you link your application with the /LARGEADDRESSAWARE flag, your application will be able to allocate 3GB of memory (if XP is booted with the /3GB flag, for Vista/W7 there's probably a similar flag but I don't know this). On 64-bit Operating systems, a LargeAddressAware executable can even use up to 4 GB of memory.