Ah, you've hit on the crux of memory management in managed code: if you're an object, and you have a reference to another object (even if it's only in the form of an event listener), then you are at least one reason that object won't be removed from memory during a GC.
For display objects, and in my experience pretty much anytime you want to subscribe to an event dispatcher but not be responsible for that dispatcher's remaining in memory, you should add your event listener with the weak reference option:
myPublisher.addEventListener("myEvent", myHandlerFunction, false, 0, true);
In just about every situation I encounter these days, "false, 0, true" (where true means "use weak reference," and which translates loosely as "add this listener, but don't make it a reason for the dispatcher not to get cleared from memory" -- see the docs for more information) is the proper way to add event listeners. Very few textbooks or documentation snippets illustrate this approach for some reason, which is unfortunate, because it makes for a much more intuitive memory-management experience. I'd suggest using it as the rule rather than the exception.
Hope it helps!