Excessive use of subreports can increase resource demands for a report. Each subreport spawns its own thread during fill time. Also if you are using scriptlets or helpers classes ensure all resources they create are being properly cleaned up. Also ensure you are using the latest JR package.
One note. The JR api includes a set of virtualizers. When the report is filling, the entire filled report is created as an object (JasperPrint). Depending on the numbers of pages in the report, this object can get pretty large. The virtualizers can be configured to write to the file system when a set page-threshold has been reached.
Typically this reduces overall memory usage but increases fill time. A comprise might be the gzip virtualizer which rather than writing to the file system, compresses the generated objects using gzip.
Here is an FAQ on the subject:
http://jasperforge.org/uploads/publish/jasperreportswebsite/trunk/faq.html?group_id=252#FAQ13
The example it refers to can be obtained by downloading the JR source.
http://sourceforge.net/project/showfiles.php?group_id=36382&package_id=28579
From a high level view, if you are running this report in an application you can instantiate the virtualizer (here is the interface listing the known implementing classes):
http://jasperreports.sourceforge.net/api/net/sf/jasperreports/engine/JRVirtualizer.html
Then the reference is used at fill time:
JRDataSource ds = new JREmptyDataSource(10);
JRFileVirtualizer virtualizer = new JRFileVirtualizer(2, "tmp");
JasperPrint jasperPrint = fillReport(fileName, ds, virtualizer);
If you are in iReport a virtualizer can be configured in Options -> Settings.
I hope this is helpful.
Luke