Well, as a first point of call, I'd say of the common metrics HSV (Hue, Saturation and Value) or HSL are better representative of how humans perceive colour than say RGB or CYMK. See HSL, HSV on Wikipedia.
I suppose naively I would plot the points in the HSL space for the two colours and calculate the magnitude of the difference vector. However this would mean that bright yellow and bright green would be considered just as different as green to dark green. But then many consider red and pink two different colours.
Moreover, difference vectors in the same direction in this parameter space are not equal. For instance, the human eye picks up green much better than other colours. A shift in hue from green by the same amount as a shift from red may seem greater. Also a shift in saturation from a small amount to zero is the difference between grey and pink, elsewhere the shift would be the difference between two shades of red.
From a programmers point of view, you would need to plot the difference vectors but modified by a proportionality matrix that would adjust the lengths accordingly in various regions of the HSL space - this would be fairly arbitrary and would be based on various colour theory ideas but be tweaked fairly arbitrarily depending on what you wanted to apply this to.
Even better, you could see if anyone has already done such a thing online...