C++ & DirectX - geometry question

Answer 1

I am surprised that std::map is performing so bad. It might be worthwile ask a question (search for existing answers first!) specifically about std::map performance on pointers.

Given that Geometry and CachedGeometry are objects that you control, you can do whatever you want to maintain links between them. One way is to make the link 2-way: Both Geometry and CachedGeometry have pointers to each other, and if CachedGeomitry is destroyed, it tells Geometry to null the reference to it's CachedGeometry. If your app is single threaded this could be very simple. If not, it is still doable but will require you to think how you handle(or prevent) situations when an object is deleted in mid air.