Looking for a "build storage filesystem" that can merge duplicate and similar files for increased storage space

Question

Our shop is constantly running out of disk space, because we have a mandate from the developers and management to keep all of the nightly builds, as it greatly aids debugging.

Each build generally has few changes. However, almost every file is different from one build to the next, because the compiler and linker insert date/time stamps (and versions) in the objects and executables.

So the question is: does anyone know of a file system that not only can "merge duplicate files", but can also "merge similar files", since these files are several KB to MB, but only a few bytes per file have changed? (Even with code changes, the impact on the executables will generally be small, unless a major header or library is changed, etc.)

Answer 1

Why not a traditional version control system?

You begin you the first build ever and then commit each other build in succession on top of that. If a file is a duplicate, no delta will be created. If it has modification, it will store only the change.

Ain't that what you asked for?

Answer 2

I think cramfs does this, but creating cramfs images (which are read-only) may be a headache.

Hmm.. Why would you need to archive builds? Shouldn't it be trivial to recreate a build from a previous source-controlled version?

Answer 3

Generally my opinion on these type of topics is "don't try to code your way out of a poor management decision".

If they want to keep that many old build files, "man up" and buy the space (disk space is cheap).

If they don't want to buy the disks, "man up" and pick a reasonable number of builds to keep that fits within the available space.

It isn't really that hard of a problem.

The farthest I would go down this path would be to enable "disk compression" on the drive if it's available, since that is handled by the OS and doesn't need an additional app and the support that goes with it. Sure, it's a small performance hit - but it's simple.

Save the "development" juice for the important things. :-)

Ron

Answer 4

You most likely need a filesystem or volume manager with transparent compression. Performance would naturally be hurt, but if you use light compression it shouldn't be too much of a problem.

Answer 5

For a fraction of the effort of installing and maintaining a new file management scheme, I would buy more hard drives. I was at Frys last night and they were selling a 1 TB external drive for $100; it's a USB drive, all you need to do is plug it in. If it gets filled up, just buy another one. Speaking as a developer, I'd be surprised if your shop really wants to hold on to the builds indefinitely -- I suspect the value of the nightly builds would be low after some number of months.

This is not an elegant solution but it may be more cost effective.

Answer 6

This is a poor man's solution, but if build storage is a major financial decision, then I think this answer is warranted. :)

First, it is highly unlikely that you will find a brand new storage/source control system that does this type of complex file referencing for less than you could just pay for new storage.

How about keeping the last 2 months worth of builds on expensive storage (i.e. RAID) and then go get yourself some cheap storage for archiving (i.e. a 1TB USB drive that someone else mentioned).

Write a simple little console app that runs as a scheduled task every night and moves all files < 2 months ago to the archive drive.

Answer 7

I think you should use an application to delete duplicate files that will be a better solution for you, i use a software to delete duplicate files called duplicate finder 2009.

Answer 8

Opensolaris and ZFS.

http://blogs.sun.com/bonwick/entry/zfs%5Fdedup