The 32-bit/64-bit part is unrelated to Java
It turns out that memory locations in a 32-bit system are referenced by 32-bit unsigned integers. This allows up to 2^32 possible memory locations. Since each location stores 1 byte you get 2^32 bytes or 4 GB if you prefer.
On a 64 bit system there are 2^64 locations, or 16 exabytes.
Now, in Windows, the contiguous part becomes a big issue, but that is just how Windows does things. The idea is that you need to have an entire "uninterrupted" range for your heap. Sadly, Windows allocates some memory somewhere in the middle. This basically leaves you with about half the left side or half the right side, about 1.5-2GB chunks, to allocate your heap.
Check out this question for more details on 32 vs 64 bit.
Edit: Thanks mrjoltcola for the exa prefix!