It sounds like you really know what you're doing. You've profiled your application, and you know exactly where cycles are being used. You understand that calling the constructor to a reference counting pointer is expensive only if you do it constantly.
The only heads up I can give you is: suppose inside function f(t *ptr), if you call another function that uses shared pointers, and you do other(ptr) and other makes a shared pointer of the raw pointer. When that second shared pointers' reference count hits 0 then you have effectively deleted your object....even though you didn't want to. you said you used reference counting pointers a lot, so you have to watch out for corner cases like that.
EDIT:
You can make the destructor private, and only a friend of the shared pointer class, so that way the destructor can only be called by a shared pointer, then you're safe. Doesn't prevent multiple deletions from shared pointers. As per comment from Mat.