I think you are on the right track with the "git filter-branch" command you tried. The problem is you haven't told it to keep the file in any commits, so it is removed from all of them. Now, I don't think there is a way to directly tell git-filter-branch to skip any commits. However, since the commands are run in a shell context, it shouldn't be too difficult to use the shell to remove all but the last X number of revisions. Something like this:
KEEP=10 I=0 NUM_COMMITS=$(git rev-list master | wc -l) git filter-branch --index-filter 'if [[ ${I} -lt $((NUM_COMMITS - KEEP)) ]]; then git rm --cached --ignore-unmatch big_manual.txt; fi; I=$((I + 1))'
That would keep big_manual.txt in the last 10 commits.
That being said, like Charles has mentioned, I'm not sure this is the best approach, since you're in effect undoing the whole point of VCS by deleting old versions.
Have you already tried optimizing the git repository with 'git-gc' and/or 'git-repack'? If not, those might be worth a try.