How is the rectangle represented? Three points? Four points? Point, sides and angle? Two points and a side? Something else? Without knowing that, any attempts to answer your question will have only purely academic value.
In any case, for any convex polygon (including rectangle) the test is very simple: check each edge of the polygon, assuming each edge is oriented in counterclockwise direction, and test whether the point lies to the left of the edge (in the left-hand half-plane). If all edges pass the test - the point is inside. If one fails - the point is outside.
In order to test whether the point (xp, yp)
lies on the left-hand side of the edge (x1, y1) - (x2, y2)
, you need to build the line equaition for the line containing the edge. The equation is as follows
A * x + B * y + C = 0
where
A = -(y2 - y1)
B = x2 - x1
C = -(A * x1 + B * y1)
Now all you need to do is to calculate
D = A * xp + B * yp + C
If D > 0, the point is on the left-hand side. If D < 0, the point is on the right-hand side. If D = 0, the point is on the line.
However, this test, again, works for any convex polygon, meaning that it might be too generic for a rectangle. A rectange might allow a simpler test... For example, in a rectangle (or in any other parallelogram) the values of A
and B
have the same maginitude but different signs for opposing (i.e. parallel) edges, which can be exploited to simplify the test.