First you have to choose th appropriate color space you want the color comparisons to occur in (RGB, HSV, HSL, CMYK, etc.).
Assuming you want to know how close two points in the 3-dimenionsal RGB space are to each other, you can calculate the Pythagorean distance between them, i.e.,
d2 = (r1 - r2)**2 + (g1 - g2)**2 + (b1 - b2)**2;
This actually gives you the square of the distance. (Taking the square root is not necessary if you're comparing only the squared values.)
This assumes that you want to treat the R, G, and B values equally. If you'd rather weight the individual color components, such as what happens when you convert RGB into grayscale, you have to add a coefficient to each term of the distance, i.e.,
d2 = 30*(r1-r2)**2 + 59*(g1-g2)**2 + 11*(b1-b2)**2;
This assumes the popular conversion from RGB to grayscale of 30% red + 59% green + 11% blue.
Update
That last equation should probably be
d2 = (30*(r1-r2))**2 + (59*(g1-g2))**2 + (11*(b1-b2))**2;