Because newer versions of Rails use Rack under the hood, it should be fairly easy to switch between web servers. When simply starting Unicorn with your application in a development environment, what specific issues are you encountering?
You should be able to keep using Nginx as a reverse proxy. From what I've gathered, Unicorn is supposed to do the load balancing across workers itself, because it only has a single listening socket/port it shares across them. Thus Nginx should normally be configured as a very simple reverse proxy and asset server, without load balancing. (Except, of course, when spreading workers across machines.)
Is there a specific dependence on Passenger within your application that you're hoping to solve? Or are you asking for generic deployment instructions for Unicorn?