NSOutlineView
is a sublcass of NSTableView
. So an outline view also has columns, each of which has its own cell associated with it. The outline view will use the cell associated with each column to draw its content. So, if you want to change the font for the column, you'll need to change the font of the cell.
In your NIB file window in Interface Builder, switch to the tree view. In the tree view, expand the outline view and each of its table columns to locate the cells. Go ahead and selet the text field cell for one of the columns. Now you can use the inspector to set the size of the cell to Small. (I'd recommend against setting the font directly; if the "Small" font changes in a future release of Mac OS X (unlikely, but hey, stranger things have happened) you won't have to do anything to get the new font.) You'll need to repeat this step for each column in your outline view.
You'll also need to change the row height in your outline view. Interface Builder isn't smart enough to see that all your cells are set to the "Small" control size, so you'll have to set this one by hand.
Hope that points you in the right direction.