The real question is why you don't know whether you're observing it or not.
If you're doing this in the class of the object being observed, stop. Whatever's observing it expects to keep observing it. If you cut off the observer's notifications without its knowledge, expect things to break; more specifically, expect the observer's state to go stale as it doesn't receive updates from the formerly-observed object.
If you're doing this in the observing object's class, simply remember which objects you're observing (or, if you only ever observe one object, whether you're observing it). This is assuming that the observation is dynamic and between two otherwise-unrelated objects; if the observer owns the observed, just add the observer after you create or retain the observed, and remove the observer before you release the observed.
Adding and removing an object as an observer should usually happen in the observer's class, and never in the observed object's.