+3  A: 

You might try simply having two outline views: One of fixed height, pinned to the bottom of their superview, and the other of variable height, with its bottom immediately above the top of the first. The fixed-height outline view would contain those Logbook and Trash items, and the variable-height outline view would contain all the others.

The tricky part would be making this play nice with a scroll view, but I think you could do it. I imagine you'd put them both in a fully-resizable NSView and make that the scroll view's document view.

Peter Hosey
+1. I've seen other lists (like the Login Options item in the Accounts prefpane) replicated in this way.
iKenndac
However, you should also manage not only scrolling states and 2 views but indentation level, next responders, some hot-keys etc. Question is still open :)
Stream
Indentation level shouldn't change if those are top-level items. Good point about the responder chain, though.
Peter Hosey
Considering indentation levels. Please take a look at the ‘Music Videos’ item before and after adding new entry to the ‘test’ folder: http://bit.ly/1U5gkL
Stream
Ooh. Yeah, you're right.
Peter Hosey
+1  A: 

I've decided to hardcode the solution by adding 8 pixels of height for every root item in group style. So, the code looks like this:

- (CGFloat)outlineView:(NSOutlineView *)ov heightOfRowByItem:(id)item;
{
    if (![item isSpacer]) return [ov rowHeight];

    static const CGFloat ADDITIONAL_SPACE = 8.0f;
    NSUInteger numberOfRootGroups = 2;
    CGFloat heightOfRows = [ov rowHeight] * ([ov rowForItem:item] + 1) 
        + ADDITIONAL_SPACE * numberOfRootGroups;
    CGFloat heightOfSidebar = [[ov superview] frame].size.height;
    return MAX(0.0f, heightOfSidebar - heightOfRows);
}

Thanks to everybody for support!

Stream