If you have problems determining the correct way (place, order and so on) of destroying standard objects in a Delphi program, then using reference counted objects or interfaces instead will not help you at all.
I understand that you want the nodes in a graph to keep references to each other, and when there are no references left to an object, it should be destroyed automatically. But consider the fact that two nodes can each have a reference to the other node, and the ref count will never reach 0 again, so that those objects will never be freed. You will need to break at least one of the references / dependencies manually. And if you have to do this anyway, then you can as well skip reference counting altogether. For some more information see the Wikipedia article about weak references. Interfaces can be used in Delphi for reference counting, but weak references can only be maintained with clever typecasting. For an example see this source code and its comments.
One idea that you might want to explore is to keep the standard lifetime management for your objects, and let the graph objects keep track of the dependencies instead.
Let each graph object have a list of other objects that it has connections with. Now you can simply free any of the objects, and the housekeeping code for the list will remove all references to the object being destroyed from all other objects. If you want to modify the graph you simply free the nodes that you want removed, and the dependencies will be updated. If you want to destroy the whole graph, just destroy all nodes. Basically you have a list of nodes (ownership, lifetime management) and another data structure for describing the graph.