My program is fast enough, but I'd rather give up that speed for memory optimization since one user's maximum memory usage goes up to 300 MB meaning few of them could constantly crash the application. Most of the answers I found were related to speed optimization, and other were just general ("if you write directly from a database to memory there shouldn't be much memory usage"). Well, it seems there is :) I was thinking about not posting code so I wouldn't "lock" someone's ideas, but on the other hand, I could be wasting your time if you don't see what I've already done so here it is:
// First I get the data from the database in a way that I think can't be more
// optimized since i've done some testing and it seems to me that the problem
// isn't in the RS and setting FetchSize and/or direction does not help.
public static void generateAndWriteXML(String query, String oznaka, BufferedOutputStream bos, Connection conn)
throws Exception
{
ResultSet rs = null;
Statement stmt = null;
try
{
stmt = conn.createStatement(ResultSet.TYPE_SCROLL_INSENSITIVE, ResultSet.CONCUR_READ_ONLY);
rs = stmt.executeQuery(query);
writeToZip(rs, oznaka, bos);
} finally
{
ConnectionManager.close(rs, stmt, conn);
}
}
// then I open up my streams. In the next method I'll generate an XML from the
// ResultSet and I want that XML to be saved in an XML, but since its size takes up
// to 300MB, I want it to be saved in a ZIP. I'm thinking that maybe by writing
// first to file, then to zip I could get a slower but more efficient program.
private static void writeToZip(ResultSet rs, String oznaka, BufferedOutputStream bos)
throws SAXException, SQLException, IOException
{
ZipEntry ze = new ZipEntry(oznaka + ".xml");
ZipOutputStream zos = new ZipOutputStream(bos);
zos.putNextEntry(ze);
OutputStreamWriter writer = new OutputStreamWriter(zos, "UTF8");
writeXMLToWriter(rs, writer);
try
{
writer.close();
} catch (IOException e)
{
}
try
{
zos.closeEntry();
} catch (IOException e)
{
}
try
{
zos.flush();
} catch (IOException e)
{
}
try
{
bos.close();
} catch (IOException e)
{
}
}
// And finally, the method that does the actual generating and writing.
// This is the second point I think I could do the memory optimization since the
// DataWriter is custom and it extends a custom XMLWriter that extends the standard
// org.xml.sax.helpers.XMLFilterImpl I've tried with flushing at points in program,
// but the memory that is occupied remains the same, it only takes longer.
public static void writeXMLToWriter(ResultSet rs, Writer writer) throws SAXException, SQLException, IOException
{
//Set up XML
DataWriter w = new DataWriter(writer);
w.startDocument();
w.setIndentStep(2);
w.startElement(startingXMLElement);
// Get the metadata
ResultSetMetaData meta = rs.getMetaData();
int count = meta.getColumnCount();
// Iterate over the set
while (rs.next())
{
w.startElement(rowElement);
for (int i = 0; i < count; i++)
{
Object ob = rs.getObject(i + 1);
if (rs.wasNull())
{
ob = null;
}
// XML elements are repeated so they could benefit from caching
String colName = meta.getColumnLabel(i + 1).intern();
if (ob != null)
{
if (ob instanceof Timestamp)
{
w.dataElement(colName, Util.formatDate((Timestamp) ob, dateFormat));
}
else if (ob instanceof BigDecimal)
{
// Possible benefit from writing ints as strings and interning them
w.dataElement(colName, Util.transformToHTML(new Integer(((BigDecimal) ob).intValue())));
}
else
{ // there's enough of data that's repeated to validate the use of interning
w.dataElement(colName, ob.toString().intern());
}
}
else
{
w.emptyElement(colName);
}
}
w.endElement(rowElement);
}
w.endElement(startingXMLElement);
w.endDocument();
}
EDIT: Here is an example of memory usage (taken with visualVM):
EDIT2: The database is Oracle 10.2.0.4. and I've set ResultSet.TYPE_FORWARD_ONLY and got a maximum of 50MB usage! As I said in the comments, I'll keep an eye on this, but it's really promising.
EDIT3: It seems there's another possible optimization available. As I said, I'm generating an XML, meaning lots of data is repeated (if nothing else, then tags), meaning String.intern() could help me here, I'll post back when I test this.
