Interesting. What happens when you set max memory to 513M?
If that fails, it's possibly a problem I haven't seen in quite a while. An ancient COBOL compiler I used refused to work on PCs with 640K of RAM because they used a signed-number check to decide if there was enough memory.
And in that world, 640K actually had the high bit set, hence was a negative number, so the check always failed.
I find it hard to believe that this would be the case in todays world but it may be worth looking at.
If it doesn't fail at 513M, then it may just be that you're trying to allocate too much memory. It's not necessarily physical memory that matters, address space may be the issue but you should have 2G (at least) of that as well in 32bit Windows.
With your settings shown, you use 1G just for permgen and heap. Try adjusting these until it works and post the figures you have.