OSGi pays off because it enforces modularization at runtime, something you previously did not have, often causing the design on paper and implementation to drift apart. This can be a big win during development.
It definitely helps make it easier to work in teams, if you let teams focus on a single module (possibly a set of bundles), and if you get your modularization right. One could argue that one can do the same thing with a build tool like Ant+Ivy or Maven and dependencies, the granularity of dependencies that OSGi uses is in my opinion far superior, not causing the typical "dragging in everything plus the kitchen sink" that JAR level dependencies cause.
Modular code with less dependencies tends to lead to cleaner and less code, in turn leading to less bugs that are easier to test for and solve. It also promotes designing components as simple and straightforward as possible, whilst at the same time having the option to plug in more complicated implementations, or adding aspects such as caching as separate components.
Hot deployment, even if you do not use it at runtime, is a very good test to validate if you modularized your application correctly. If you cannot start your bundles in a random order at all, you should investigate why. Also, it can make your development cycle a lot quicker if you can update an arbitrary bundle.
As long as you can manage your modules and dependencies, big projects stay manageable and can be easily evolved (saving you from the arguably bad "complete rewrite").
The downside of OSGi? It's a very low-level framework, and whilst it solves the problems it is intended for quite well, there are things that you still need to resolve yourself. Especially if you come from a Java EE environment, where you get free thread-safety and some other concepts that can be quite useful if you need them, you need to come up with solutions for these in OSGi yourself.
A common pitfall is to not use abstractions on top of OSGi to make this easier for the average developer. Never ever let them mess with ServiceListeners or ServiceTrackers manually. Carefully consider what bundles are and are not allowed to do: Are you willing to give developers access to the BundleContext or do you hide all of this from them by using some form of declarative model.