You don't say how much memory your machine actually has (and whether all of it is available to the Oracle instance).
There are two types of memory used by Oracle. The first is the shared memory (SGA), the second is process memory (PGA). SGA is generally easier to control.
From SQL*Plus, try SHOW SGA. See if that matches your target.
Also try SHOW PARAMETER PGA
Then query v$process and compare the total PGA of the processes with the target.
You may have set the PGA_AGGREGATE_TARGET/SGA_MAX_SIZE total too high.
Or maybe Oracle can't keep the PGA memory down to the target (eg due to sessions storing lots of information in PL/SQL arrays).