Found this in the documentation:
The displayed resolution of an image item is dependent on the sizeMode of the toolbar. You should provide image representations specific to the default, regular and small size modes in a single image that supports multiple image representations such as icns or tiff. The appropriate image representation is automatically displayed for the toolbar's current sizeMode. If an appropriate representation is not available, the toolbar scales the representation to the appropriate size for the current mode, at a cost in performance and appearance. Images that are not square are scaled to fit. An image item’s image is also scaled down and used in the image item’s overflow menu item. (Emphasis added)
This seems you might want to try fiddling with the NSToolbarItem's minSize and maxSize properties.