How to recover Git objects damaged by hard disk failure?

Question

I have had a hard disk failure which resulted in some files of a Git repository getting damaged. When running git fsck --full I get the following output:

error: .git/objects/pack/pack-6863e0a0e4b4ded6090fac5d12eba6ca7346b19c.pack SHA1 checksum mismatch
error: index CRC mismatch for object 6c8cae4994b5ec7891ccb1527d30634997a978ee from .git/objects/pack/pack-6863e0a0e4b4ded6090fac5d12eba6ca7346b19c.pack at offset 97824129
error: inflate: data stream error (invalid code lengths set)
error: cannot unpack 6c8cae4994b5ec7891ccb1527d30634997a978ee from .git/objects/pack/pack-6863e0a0e4b4ded6090fac5d12eba6ca7346b19c.pack at offset 97824129
error: inflate: data stream error (invalid stored block lengths)
error: failed to read object 0dcf6723cc69cc7f91d4a7432d0f1a1f05e77eaa at offset 276988017 from .git/objects/pack/pack-6863e0a0e4b4ded6090fac5d12eba6ca7346b19c.pack
fatal: object 0dcf6723cc69cc7f91d4a7432d0f1a1f05e77eaa is corrupted

I have backups of the repository, but the only backup that includes the pack file has it already damaged. So I think that I have to find out a way to retrieve the single objects from different backups and somehow instruct Git to produce a new pack with only correct objects.

Can you please give me hints how to fix my repository?

Answer 1

Git checkout can actually pick out individual files from a revision. Just give it the commit hash and the file name. More detailed info here.

I guess the easiest way to fix this safely is to revert to the newest uncommited backup and then selectively pick out uncorrupted files from newer commits. Good luck!

Answer 2

In some previous backups, your bad objects may have been packed in different files o may be loose objects yet. So your objects may be recovered.

It seems there are a few bad objects in your database. So you could do in the manual way.

Because of 'git hash-object', 'git mktree' and 'git commit-tree' do not write the objects because they are found in the pack, then start doing this:

mv .git/objects/pack/* <somewhere>
for i in <somewhere>/*.pack; do
  git unpack-objects -r < $i
end
rm <somewhere>/*

(Your packs are moved out from the repository, and unpacked again in it; only the good objects are now in the database)

You can do:

git cat-file -t 6c8cae4994b5ec7891ccb1527d30634997a978ee

and check the type of the object.

If the type is blob: retrieve the contents of the file from previous backups (with 'git show' or 'git cat-file' or 'git unpack-file'; then you may 'git hash-object -w' to rewrite the object in your current repository.

If the type is tree: you could use 'git ls-tree' to recover the tree from previous backups; then 'git mktree' to write it again in your current repository.

If the type is commit: the same with 'git show', 'git cat-file' and 'git commit-tree'.

Of course, I would backup your original working copy before start this process.

Also, take a look to here

Answer 3

Banengusk was putting me on the right track. For further reference, I want to post the steps I took to fix my repository corruption. I was lucky enough the find all needed objects either in older packs or in repository backups.

# Unpack last non-corrupted pack
$ mv .git/objects/pack .git/objects/pack.old
$ git unpack-objects -r < .git/objects/pack.old/pack-012066c998b2d171913aeb5bf0719fd4655fa7d0.pack
$ git log
fatal: bad object HEAD
$ cat .git/HEAD 
ref: refs/heads/master
$ ls .git/refs/heads/
$ cat packed-refs 
# pack-refs with: peeled 
aa268a069add6d71e162c4e2455c1b690079c8c1 refs/heads/master
$ git fsck --full 
error: HEAD: invalid sha1 pointer aa268a069add6d71e162c4e2455c1b690079c8c1
error: refs/heads/master does not point to a valid object!
missing blob 75405ef0e6f66e48c1ff836786ff110efa33a919
missing blob 27c4611ffbc3c32712a395910a96052a3de67c9b
dangling tree 30473f109d87f4bcde612a2b9a204c3e322cb0dc
# Copy HEAD object from backup of repository
$ cp repobackup/.git/objects/aa/268a069add6d71e162c4e2455c1b690079c8c1 .git/objects/aa
# Now copy all missing objects from backup of repository and run "git fsck --full" afterwards
# Repeat until git fsck --full only reports dangling objects
# Now garbage collect repo
$ git gc
warning: reflog of 'HEAD' references pruned commits
warning: reflog of 'refs/heads/master' references pruned commits
Counting objects: 3992, done.
Delta compression using 2 threads.
fatal: object bf1c4953c0ea4a045bf0975a916b53d247e7ca94 inconsistent object length (6093 vs 415232)
error: failed to run repack
# Check reflogs...
$ git reflog
# ...then clean
$ git reflog expire --expire=0 --all
# Now garbage collect again
$ git gc       
Counting objects: 3992, done.
Delta compression using 2 threads.
Compressing objects: 100% (3970/3970), done.
Writing objects: 100% (3992/3992), done.
Total 3992 (delta 2060), reused 0 (delta 0)
Removing duplicate objects: 100% (256/256), done.
# Done!