Seems to me that there's a lot of things happening at the same time to keep proper track of.
I would try to separate the "disposal of objects" concern, for example add to-be-disposed items in a queue/disposal manager of some sort which would process (call Dispose on) items in a safer and more well designed/understood and deterministic way. If anything, this should help with debugging the problem.