Is git worth for managing many files bigger than 500MB

Question

I would put under version control a big amount of data, i.e. a directory structure (with depth<=5) with hundreds files with size about 500Mb).

The things I need is a system that help me: - to detect if an files has been changed - to detect if files were added/removed - to clone the entire repository in another location - to store a "checkpoint" and restore it later

I don't need sha1 for change detect, something faster is acceptable.

Is git worth for this? There is a better alternative?

Answer 1

Maybe something like rsync is better for your needs (if you just want some backups, no concurrency, merge, branching etc.)

Answer 2

Unison File Synchroniser is an excellent tool for maintaining multiple copies of large binary files. It will do everything you ask for apart from storing a checkpoint - but that you could do with an rsync hardlink copy.

Answer 3

As I mentioned in "What are the Git limits", Git is not made to manage big files (or big binary files for that matter).

Git would be needed if you needed to:

• know what has actually changed within a file. But for the directory-level, the other answers are better (Unison or rsynch)
• keep a close proximity (i.e. "same referential") between your development data, and those large resources. Having only one referential would help, but then you would need a fork of Git, like git-bigfiles to efficiently manage them.

---

Note: still using Git, you can try this approach

Unfortunately, rsync isn't really perfect for our purposes either.

• First of all, it isn't really a version control system. If you want to store multiple revisions of the file, you have to make multiple copies, which is wasteful, or xdelta them, which is tedious (and potentially slow to reassemble, and makes it hard to prune intermediate versions), or check them into git, which will still melt down because your files are too big.
• Plus rsync really can't handle file renames properly - at all.

Okay, what about another idea: let's split the file into chunks, and check each of those blocks into git separately.
Then git's delta compression won't have too much to chew on at a time, and we only have to send modified blocks...

Based on gzip --rsyncable, with a POC available in this Git repo.

Answer 4

If you're on a unix system (probably are, since you're using git):

• Use a git repo for all the small stuff.
• Symlink large files from a single "large_files" folder to the appropriate locations within your repository.
• Backup the large_files folder using a more traditional-non-versioning backup system, bundle 'em all up into a zip file from time to time if you need to pass 'em to others.

That way, you get the benefits of git, you keep whatever tree structure you want, and the large sized files are backed up elsewhere, despite appearing to still be inside the normal folder hierarchy.