I'm running Linux, so you'll have to adapt this answer somewhat, but it should be applicable.
In the home directory that the installer jar creates, you'll find a file called "squirrel-sql.sh". The last line of this file is the java launch cmd. Not sure about the mac launch configuration, you might need to look for the file in this directory that contains this line. Mine reads
$JAVA -Xmx256m -cp $TMP_CP net.sourceforge.squirrel_sql.client.Main --log-config-file $SQUIRREL_SQL_HOME/log4j.properties --squirrel-home $SQUIRREL_SQL_HOME $1 $2 $3 $4 $5 $6 $7 $8 $9
The -Xmx256m configures the heap - it sets the max heap size to 256m. Just edit the file to read -Xmx512m or whatever. (for a list of the memory flags, type "java -X" at a command prompt).