Jon Skeet is right. You need to use a system with Hue as a component instead of RGB if you are concerned about the color component of the match differing too much. HSL or HSV will both work fine for this purpose.
Then you need to fiddle with the distance formula to weight up hue until you are happy with the results. Note that you will find the problem is actually essentially insoluble unless you have a large number of colors to match against or your input colors are confined to a small range of possible values. This is because although it might seem you can pin any color you want to one of 8 (red, orange yellow, green, blue, violet, black and white) or one of 16, in reality you'll find your algorithm will always find what seem to be obviously incorrect matches because with 3 axis of movement (hue, saturation, value or red, green, blue), there are a lot more "basic" colors than you might think at first glance.