Hardware Lossless Compression for Hard Drives?

Answer 1

In the world of 1TB drives for $100 speed is a much more valuable resource than space. It wouldn't be worth it.

EDIT:

Ah so you're saying that it would be quicker to grab 100 bytes of compressed data off the platters, decompress it and then send it along to the system rather than grabbing 800 bytes of uncompressed data and send it along to the system because seek times are so slow.

That seems like a clever approach but I'm willing to bet that if the trade off ends up being worth it that hard drive manufacturers have already employed this technique and hard drive speeds are what they are in spite of the fact.

But who knows, you may be on to something!

Answer 2

It's possible, but it's very, very tricky. You'll have to develop custom drivers because while encrypted sectors are the same size as regular sectors, and thus use the same math to find data, compressed sectors are "smaller" so you either have to hold a map of 'real' sectors to compressed sectors in the OS, or on the drive itself.

The only other aspect is speed of access and latency. It shouldn't affect seek, but it may take longer to compress data than it would to write it - compression is fairly compute intensive.

Further, compression isn't really good until you get to large chunks of data. You can probably compress 512 bytes (1 sector) on the fly and get a few percent compression on average, but people really want to see 20% and more compression before they're willing to pony up the extra cash for the hardware.

It will require more on disk processing power and memory, which will increase the cost of the drive.

Further, drive capacity is growing at a rate that it's probably not cost effective to do this.

Let's say, for instance, that you develop the miracle compression that doubles the space with no performance drop, no extra (flaky or crash prone) drivers, works on any OS, etc. But it adds $100 to the cost of the drive.

It might make sense for someone to do this for a 1TB drive now, converting it into a 2TB drive, but in 6-8 months the 2TB drives will be below $200. It won't be worth it for any smaller drive, because you can get a 1TB for $99 now.

If you made it so it would work between the drive and the computer you're going to have much greater latency than building it directly into the drive, and the price/performance hit might not make it worthwhile.

So, technically it's possible, but has pitfalls and adds complexity and points of weakness to the system, but even if it didn't have these drawbacks, it is likely that it isn't worthwhile.

Answer 3

Who remembers Stacker? This was all already done-to-death in the '80's/90's. Speed was never a problem, and neither was it "tricky." It's just completely unnecessary these days.

Answer 4

As previously said, the gain is not that big, especially if you are storing seldom accessed files in a compressed form anyway.

As it would be hard to do in hardware (What disk size should be reported? What do you do if the entropy of the input equals its size?) and modern CPUs+RAM are blazingly fast compared to HDDs anyway, just do it in software.

An implementation I know of is compFUSed which is layered on top of any other file system, another one is ZFS Blog entry about how to enable it which supports compression natively.

Answer 5

I remember about 15 years ago seeing an advertisement for an IDE controller card that would do hardware compression. Not sure if it was any good or not. Those were the days when 1GB drives where over $1,000.

Answer 6

I had also thought of this idea a while ago for network traffic- which has been done before: there are accelerator cards for comressing using gzip: http://www.aha.com/show_prod.php?id=36

I had also thought of another benefit would be that now you can transfer without compressing the contents from the drive- simply read from the disk the compressed blocks and send rather than having to compress at that later time.