It'll be 'safe', in that you won't break the repository or your working copies or any future functionality.
But I like the idea of keeping them, simply because you could in theory go and work out who actually did a particular piece of work and what its real history was (in theory, if you cared enough). If you delete the merge info, you no-longer have any hint where the associated code came from.
However, it's a weak argument for keeping this info if it's getting out of hand and causing issues, or even if it's just annoying you.
Perhaps you could keep it for a couple of revisions then get rid of 'old' ones. That way, a blame will turn up the merge revision, which will also allow you to find the associated merge info as it was then.