Find the slope a1 of AB by dividing the y-difference with the x-difference; then draw a perpendicular line (with slope a2 = -1/a1, you need to solve for the offset (b2) by putting P's coordinates into y = a2*x + b2); then you have two lines (i.e. two linear equations), and you need to solve the intersection. That will be your closest point.
Do the math right, and the function will be pretty trivial to write.
To elaborate a bit:
Original line:
y = a1 * x + b1
a1 = (By - Ay) / (Bx - Ax) <--
b1 = Ay - a1 * Ax <--
Perpendicular line:
y = a2 * x + b2
a2 = -1/a1 <--
b2 = Py - a2 * Px <--
Now you have P which lies on both lines:
y = a1 * x + b1
y = a2 * x + b2
--------------- subtract:
0 = (a1 - a2) * Px + (b1 - b2)
x = - (b1 - b2) / (a1 - a2) <--
y = a1 * x + b1 <--
Hope I didn't mess up somewhere :) UPDATE Of course I did. Serve me right for not working things out on paper first. I deserved every downvote, but I'd've expected someone to correct me. Fixed (I hope).
Arrows point the way.
UPDATE Ah, the corner cases. Yeah, some languages don't handle infinities well. I did say the solution was language-free...
You can check the special cases, they're quite easy. The first one is when the x difference is 0. That means the line is vertical, and the closest point is on a horizontal perpendicular. Thus, x = Ax, y = Px
.
The second one is when y difference is 0, and the opposite is true. Thus, x = Px, y = Ay