I'm going to interpret this as simply as possible, if I'm not answering your question, I apologize.
When you really get the answer to this question, it's your first step to really thinking Object Orientedly.
In OO, when two objects both "has a" 'nother object, it's perfectly acceptable for both to refer to that other object. The trick with OO is that objects have a life of their own, they are fluid and anyone that needs them can keep a reference to them. The object must keep itself "Valid" and maintain stability when being used by many other objects. (This is why immutable objects like String are so great, they are ALWAYS as valid as the second they were created)
The one exception is if you are coding in C++ because then you actually have to manually free objects, that implies an owner that can monitor each object's lifecycle--that makes it really really hard to "Think" in OO in C++.
[Addition] Since you are referring to pointers, I'm thinking you are programming in C++ which is different. In that case, you are right. Make one manager "Own" the lifecycle of your shared object. It must not let that object die until all other references have gone away.
You can also use reference counting. Whenever someone gets a reference to your object, it calls "addReference" or something, whenever it's done it removes the reference. If anyone calls removeReference when the count is 1, the object can clean itself up. That's probably as close as you can come to true OO-style allocation/freeing in C++. It's very error-prone though.
There are libraries to do this kind of stuff I believe.