This article is a good place to start. Essentially, you instantiate some owner (that holds the outlets/actions) if it doesn't exist and ask NSBundle to load the nib with that given owner. If all goes well, you'll have your UI all wired up and ready to go.
As for how to toggle between them, this is far too broad to answer. At its most basic, you could always add the custom / alternative views as subviews of individual tab view items in a tabless tab view, asking it to select a given tab when some action is fired. A bit more complicated (only a little bit) would be to use NSView's -replaceSubview:with: to swap them out of some containing superview.
Regarding your third question ... can you please rephrase it? As you phrased it, the differences are a) you'll have more work to do loading it from a separate nib and, b) you won't be able to reuse it (ie, create multiple instances of the same subview assembly), which may or may not be important to you in this case since it seems like you'll only have one instance of the contents of CustomView and Alternative.