Once an instance of an OSGi service is
retrieved from the bundle context does
it become invalidated when the service
is stopped?
No, the reference itself does not become invalidated. As long as something within the container is holding it, it can also not be GC'ed.
However, whether or not it will be still useful, depends only on the service implementation itself, not the container.
My initial tests show that the service instance can be used even after the service bundle >is stopped, which contradicts my understanding of the dynamic nature of OSGi.
The specifications itself indicate that such references shouldn't be held, but it's up to the implementer to take care for implementing the specifications correctly; means there is no contradiction, there is only the fact that you can implement and deploy bundles which don't behave correctly according to the specifications.
I suppose this boils down to what retrieving a service (via ServiceTracker) from another bundle
in the OSGi container actually does, does it create a new instance or does it give you a pointer
to the instance that is registered in the container?
The container does not create new instances of services, except if there is a ServiceFactory involved (see specs). Looking up a service should always give you a pointer to the registered instance in the container.
Are there any dangers in using the service instance after the service has been stopped?
That only depends on the service implementation; however, by doing so in a bundle, you automatically create a bundle that is not conform with the specs.
In practice, many services are implemented to release resources and references and will not respond properly any longer.