This isn't really a drawing question so much as a general custom views question. It's important to make that distinction.
I'll describe this in terms of rectangles to give you the general idea*. You should make sure you understand the view hierarchy and view geometry in Cocoa. Without this important requisite knowledge, you'll remain dead in the water.
It's easy to set an NSButton's image, so I'll leave that to you. Your button's action, however, would tell some controller to show the "image picker" for the given button. Your image picker would be some type of borderless window with an image list inside. The image picker could be an IKImageBrowserView (you'll have to enable Image Kit in Interface Builder for this control to appear), which gives you an iPhoto-like grid of images (with/without titles, different border types, etc.).
An explanation of the operation of this controller and how it creates the window, manages the selection, and sets the button's image is very broad so if you get hung up on any of those steps, you'll need to create a separate question for each problem, otherwise this answer would have to be an instruction manual for writing your app for you.
* Your problem is a little more difficult because of your desire to have differently-shaped "popup windows" ... you'd have to make sure your available photos fit neatly within the shape so none of them are cut off. Armed with the basic knowledge of view geometry, I'll leave this to you as an exercise. A hint: you can use a borderless, transparent window to host a view that draws itself in any shape you please.