Can I use less memory when using DeflateStream?

Question

My application needs to decompress files which contain a lot of Deflate compressed blocks (as well as other types of compression and encryption). Memory profiling shows that the deflate stream constructor is responsible for allocating the majority of the application's memory over its lifetime (54.19%, followed by DeflateStream.read at 12.96%, and everything else under 2%).

To put it in perspective, each file block is usually 4KiB (decompressed) and DeflateStream's constructor allocates slightly more than 32KiB (presumably for the sliding window). The garbage collector has a field day as all these deflate streams last for almost no time at all (each goes away before the next one comes in)! Goodbye cache efficiency.

I can keep using DeflateStream, but I am wondering if there's a better alternative. Maybe a way to reset the stream and use it again?

Answer 1

Do you have any actual performance problems, or is it that you are just worried about the memory usage?

Most objects are short lived, so the memory management and the garbage collector is built to handle short lived objects efficiently. A lot of classes in the framework is designed to be used once and then thrown away to be more short lived.

If you try to hang on to objects they are more likely to survive a garbage collection, which means that they will be moved from one heap generation to another. The heap generations is not just a logical division of object, but the object is actually moved from one memory area to another. The garbage collector usually works with the principle of moving all the live objects in a heap to the next generation and then just empty out the heap, so it's the long lived objects that are costly, not the short lived objects.

Due to this design it's quite normal for the memory throughput to be high while the actual memory usage stays low.

Answer 2

There's a DeflateStream in DotNetZip, effectively a replacement of the built-in DeflateStream in the .NET BCL. Ionic.Zlib.DeflateStream has a tunable buffer size. I don't know if it will result in better memory efficiency in your scenario, but it may be worth a try. Here's the doc.

I did not test decompression, but rather compression. In my tests I found limited returns on expanding the buffer size beyond 4k, for the subset of data I compressed. On the other hand, you still get accurate, correct compression, although it is less effective, even if the buffer is 1024 bytes. I suppose you would see similar results in decompression.

In either case, the window size is not directly settable from the public interface. But, it is open source and you will be able to easily modify the default Wwindow size as appropriate. Also, if you think it's valuable, I could take a request to expose the window size as a settable param on the DeflateStream. I haven't exposed it because no one has asked for it. Yet?

You said you had other compression, too. If you're doing Zlib or GZip, there's a ZlibStream and a GZipStream in the DotNetZip package, too.

If you want to do Zip files, you need the full DotNetZip library (Ionic.Zip.dll, at ~400k). If you are just doing {Deflate, Zlib, GZip}Stream, then there is a Ionic.Zlib.dll, which is about 90k.

DotNetZip is free, but donations are encouraged.

Answer 3

Two comments without the benefit of any actual measurement to back this up:

• I think you'll find that the amount of time taken by the allocations (and zeroing) of these temporary buffers is negligible next to the time spent on the actual decompression.
• The fact that these buffers are highly transient means that although over the lifetime of the app it may be 50% of the memory, none of it exists simultaneously. Note that this also should not hurt the cache efficiency much... I'd imagine that most of these buffers will not outlive their use by much in the cache memory, because the pages would go stale very quickly.

In short, unless you have a measurable problem with the deflate stream (either in speed or absolute memory use), I'd just keep using it... better to use the solution you know than introduce another one that may have a whole different set of problems that are harder to deal with.