Use GZIP to boost XML-parsing

Question

Hello all,

I am facing the problem to handle many xml-files that are all greater than 70 MB. Validating and accessing them cost a lot of time. Now I am wondering if the following steps could increase my applications performence.

I can compress a 70MB xml-file in a gzip-file < 1MB. So I could hold only gzip files. Working with the data could be done like this:

• Make the gzip available with java.io.File (only the small file)
• Using for example StringBufferInputStream and GZIPInputStream to extract the conten within the RAM
• work with the content (RAM): Parse, Validate, ...
• create a String in RAM that represents the new xml-content (RAM)
• Use GZIPOutputStream to access the file system (small content again)

Can I do this or id there a misapprehension in my thoughts?

THX in advance! Hans

Answer 1

Reading 70 MB from the HD should take no more than 1-2 seconds (depending on your hardware, of course), so if you're having a delay greater than, say, 4 seconds, the bottleneck is not your HD but XML processing and anything you do with that.

Before getting into coding your gzip idea (which sounds fine), you could hardcode a sample XML to your code (yes, insert 70 MB as a single string), run your app having a nice button saying ("do it!") - or wait for user input if you're in the terminal - and see how much XML processing takes.

This approach will load your 70 megs into memory (as code) before processing, so you should be able to see how much it really takes to consume it.

After that, if you see it's processed quickly enough, the problem is clearly the HD. If not, then you should try to optimize your XML processing.