Given point P and triangle A, B, C, compute:
1. the unit normal of triange (A, B, P) - call it N1
2. the unit normal of triangle (B, C, P) - call it N2
(get the order right!)
Now think about the dot product N1*N2. if P is in the plane of the triangle, and inside the three sides, those normals should be parallel, so this dot product will be 1.0000 (or 0.999...). If P is kept in the plane but moved beyond side BC, those two normals will be opposite: N1*N2==-1. If P isn't in the plane, the dot product will be some intermediate value. Whoops, we still have a loophole - if P goes out past side CA. We need to compute one more:
3. the unit normal (C,A,P) called N3
Make these two tests (in an ideal world):
N1*N2 == 1.0 ?
N2*N3 == 1.0 ?
(testing N3*N1 is redundant) Of course, the test will have to allow some slop for the imperfections of computer arithmetic. Look for (N1*N2 > 1-epsilon) where epsilon is some small value, depending on accuracy needed and floating point types.
You may need a formula for these unit normals. Given (A,B,C) compute the cross product N =(B-A)x(C-B). Then divide by sqrt(N*N). Definitions of "dot product" and "cross product" are easy to find in textbooks and wikipedia etc. It is possible to increase performance with some algebra to about square roots.
I don't claim this is the fastest algorithm, but should work (until so