Use Process Explorer instead of Task Manager to look at the memory consumption of your application.
- Private Bytes: is the amount of memory that is used by your application.  This is also the value shown by Task Manager in the VM Size column (under XP)
- Virtual Memory Size: this is the largest address that is currently in use by your application.  This value is limited by 2 GB (for a 32-bit application in a 32-bit OS).
The different between Private Bytes and Virtual Memory Size is caused by memory fragmentation.  The only thing you can do about it is to use a memory manager that tries to reduce fragmentation, e.g. by using memory pools.
The DougLea memory manager is a free alternative.  There are also commercial memory manager.  You can also try to write your own memory manager (it's not that hard: you only have to override the new and delete operators).
There are also tricks to increase the limit of 2 GB.  If you link your application with the /LARGEADDRESSAWARE flag, your application will be able to allocate 3GB of memory (if XP is booted with the /3GB flag, for Vista/W7 there's probably a similar flag but I don't know this).  On 64-bit Operating systems, a LargeAddressAware executable can even use up to 4 GB of memory.