Update: For me, the reference document to tune a Sun virtual machine is Java SE 6 HotSpotTM Virtual Machine Garbage Collection Tuning (that will also give you a methodology).
As mentioned in a comment, the ergonomics feature does a pretty good job at tuning a modern JVM and should be tried first before to test more detailed controls.
If you don't get satisfying results, set the max Heap size and start to play with the generation sizes and more precisely the Young Generation. From the FAQ about Garbage Collection in the HotspotTM JavaTM Virtual Machine:
The young generation should be sized large enough so that short-lived objects have a chance to die before the next young generation collection. This is a tradeoff since a larger young generation will allow more time for objects to die but may also take longer to collect. Experiment with the size of the young generation to optimize the young generation collection time or the application throughput.
I'd follow the suggested approach:
The rules of thumb for server
applications are:
- First decide the maximum heap size you can afford to give the virtual
machine. Then plot your performance
metric against young generation sizes
to find the best setting.
- Note
that the maximum heap size should
always be smaller than the amount of
memory installed on the machine, to
avoid excessive page faults and
thrashing.
- If the total heap size is fixed, increasing the young generation size
requires reducing the tenured
generation size. Keep the tenured
generation large enough to hold all
the live data used by the application
at any given time, plus some amount of
slack space (10-20% or more).
- Subject to the above constraint on the tenured generation:
- Grant
plenty of memory to the young
generation.
- Increase the young
generation size as you increase the
number of processors, since allocation
can be parallelized.
Then, if the application does still not achieve the desired performance, start with a different collector (see Selecting a Collector).
Unless your application has rather strict pause time requirements, first run your application and allow the VM to select a collector. If necessary, adjust the heap size to improve performance. If the performance still does not meet your goals, then use the following guidelines as a starting point for selecting a collector.
- If the application has a small data set (up to approximately 100MB), then
- select the serial collector with
-XX:+UseSerialGC
.
- If the application will be run on a single processor and there are no pause time requirements, then
- let the VM select the collector, or
- select the serial collector with
-XX:+UseSerialGC
.
- If (a) peak application performance is the first priority and (b) there are no pause time requirements or pauses of one second or longer are acceptable, then
- let the VM select the collector, or
- select the parallel collector with
-XX:+UseParallelGC
and (optionally) enable parallel compaction with -XX:+UseParallelOldGC
.
- If response time is more important than overall throughput and garbage collection pauses must be kept shorter than approximately one second, then
- select the concurrent collector with
-XX:+UseConcMarkSweepGC
. If only one or two processors are available, consider using incremental mode, described below.
Actually, just read the whole document carefully (and be sure to understand the implications of a bad decision :) And don't forget:
If you cannot measure it, you cannot improve it. --Lord Kelvin
References
Below my initial answer that some readers might still find interesting:
Java Performance Tuning is a well known reference for Java performance and still updated. You might also want to check the blog of the Kirk Pepperdine (which is actually a contributor of the previous site).