views:

756

answers:

3

(solved, see bottom of the question body)
Looking for this for a long time now, what I have till now is:

Pretty much the same method, but both of them leave objects in pack files... Stuck.
What I tried:

git filter-branch --index-filter 'git rm --cached --ignore-unmatch file_name'
rm -Rf .git/refs/original
rm -Rf .git/logs/
git gc

Still have files in the pack, and this is how I know it:

git verify-pack -v .git/objects/pack/pack-3f8c0...bb.idx | sort -k 3 -n | tail -3

And this:

git filter-branch --index-filter "git rm -rf --cached --ignore-unmatch file_name" HEAD
rm -rf .git/refs/original/ && git reflog expire --all &&  git gc --aggressive --prune

The same...

Tried git clone trick, it removed some of the files (~3000 of them) but the largest files are still there...

I have some large legacy files in the repository, ~200M, and I really don't want them there... And I don't want to reset the repository to 0 :(

SOLUTION: This is the shortest way to get rid of the files:

  1. check .git/packed-refs - my problem was that I had there a refs/remotes/origin/master line for a remote repository, delete it, otherwise git won't remove those files
  2. (optional) git verify-pack -v .git/objects/pack/#{pack-name}.idx | sort -k 3 -n | tail -5 - to check for the largest files
  3. (optional) git rev-list --objects --all | grep a0d770a97ff0fac0be1d777b32cc67fe69eb9a98 - to check what files those are
  4. git filter-branch --index-filter 'git rm --cached --ignore-unmatch file_names' - to remove the file from all revisions
  5. rm -rf .git/refs/original/ - to remove git's backup
  6. git reflog expire --all --expire='0 days' - to expire all the loose objects
  7. (optional) git fsck --full --unreachable - to check if there are any loose objects
  8. git repack -A -d - repacking the pack
  9. git prune - to finally remove those objects
+1  A: 

See: How do I remove sensitive files from git’s history

The above will fail if the file does not exist in a rev. In that case, the '--ignore-unmatch' switch will fix it:

git filter-branch -f --index-filter 'git rm --cached --ignore-unmatch <filename>' HEAD

Then, to get all loose objects out of the repostiry:

git gc --prune='0 days ago'
Wayne Conrad
Yep, tried this one, still have the files in the pack, and the size didn't change too much...
Devenv
I just made a git sandbox and tried it. No good here, either. Let's see what I can figure out.
Wayne Conrad
Got it. See edited version.
Wayne Conrad
The one in the answer? :) It's the same as I posted, and it still leaves the file in the pack... try a git sandbox, doing git gc so it will pack the file, and then running this...
Devenv
Oh, the loose objects? See above. I'd be inclined to just let them be garbage collected in two weeks (the default for gc); killing _all_ loose objects is like emptying the trash--I lose any opportunities to get back anything I accidentally deleted.
Wayne Conrad
:) tried this one too... got rid of some of the files, but the biggest are still there...
Devenv
Drats. I thought that would do it. Do the files exist in any other branches?
Wayne Conrad
Have no other branches :) But I think that it might be that I moved the file from one dir to another once... I run the filter-branch on both paths, but that doesn't help...
Devenv
+1  A: 

You have various reasons for a still large git repo size after git gc, since it does not remove all loose objects.

I detail those reasons in "reduce the git repository size"

But one trick to test in your case would be to clone your "cleaned" Git repo and see if the clone has the appropriate size.

(' "cleaned" repo ' being the one where you did apply the filter-branch, and then gc and prune)

VonC
Yep, tested it already, and tested it again now, it reduced repository by 2k :) and the files are still there...
Devenv
What's weird is `git count-objects -v -> count: 0, size: 0, in-pack: 10021, packs: 1, size-pack: 244547, prune-packable: 0, garbage: 0`but: `git clone test1 test2 -> Checking out files: 100% (8509/8509), done`
Devenv
+2  A: 

I can't say for sure without access to your repository data, but I believe there are probably one or more packed refs still referencing old commits from before you ran git filter-branch. This would explain why git fsck --full --unreachable doesn't call the large blob an unreachable object, even though you've expired your reflog and removed the original (unpacked) refs.

Here's what I'd do (after git filter-branch and git gc have been done):

1) Make sure original refs are gone:

rm -rf .git/refs/original

2) Expire all reflog entries:

git reflog expire --all --expire='0 days'

3) Check for old packed refs

This could potentially be tricky, depending on how many packed refs you have. I don't know of any Git commands that automate this, so I think you'll have to do this manually. Make a backup of .git/packed-refs. Now edit .git/packed-refs. Check for old refs (in particular, see if it packed any of the refs from .git/refs/original). If you find any old ones that don't need to be there, delete them (remove the line for that ref).

After you finish cleaning up the packed-refs file, see if git fsck notices the unreachable objects:

git fsck --full --unreachable

If that worked, and git fsck now reports your large blob as unreachable, you can move on to the next step.

4) Repack your packed archive(s)

git repack -A -d

This will ensure that the unreachable objects get unpacked and stay unpacked.

5) Prune loose (unreachable) objects

git prune

And that should do it. Git really should have a better way to manage packed refs. Maybe there is a better way that I don't know about. In the absence of a better way, manual editing of the packed-refs file might be the only way to go.

Dan Moulding
Yey!!! I love you !The problem was in packed-refs file, there was refs/remotes/origin/master from times I was backing it up on some server... once I removed it it all began to disappear... Thank you! (updating the question body with the full solution)
Devenv