Let's say I've got an event listener function that is listening for an event that will never be called again throughout the lifespan of the program. The listening object will never need to be garbage collected.
How much memory does this use?
If it's negligible, I'd rather not remove the listener because having the removeEventListener() statement makes my code less readable.
It's negligible, unless you have thousands of them. Check out how EventDispatcher works and take a look at it's source code.
That depends entirely on how large and complex the listener is. In many cases, the memory impact is negligible, however, the object you're holding in memory may be keeping several other objects in memory. If one of them is a streaming video or something, it may be sucking on your memory, processor, and network.
You can also set useWeakReferences to true when you first add the event listeners. This makes the link between the listener and the event dispatcher weak so that the latter doesn't hold the prior in memory if it's deleted everywhere else. More on that here.
Still, it's never a good idea to leave objects in memory that are not going to be used again and there's no reason to avoid removeEventListener(). Striving for code readability before making it work correctly is never a good idea. If you're that concerned with the way your code looks, put the removeEventListener() calls inside a method called cleanupUnusedListeners() or something. Indeed, I would say that omitting it is less readable because when you're looking for the source of your memory leak, it will be harder to find the spot where you DIDN'T put a removeEventListener(). It may not be pretty but that's just the way it is, Jack.