3D modeling problem - revisited

Question

This is a follow-up to a previous question I posted, this time with a better description, I hope.

I have made a pair of glasses that have a pinhole camera attached onto them that records the person's eye movements. The camera sits below the eye and is hence looking UP at the eye. There is also a slight rotation around the Y-axis (i.e. the camera has a slight left- or right-wards slant) and the Z-axis (i.e the camera is slightly twisted clock-wise or anti-clockwise).

From the video, I extract a set of points in the image, that should appear to be roughly linear but do not because the camera is not oriented to look straight at the person's eye.

If I know of a unit vector U that describes the camera orientation as well as the image point IP1(x,y) where the camera is looking, how can I work out the coordinates of these image points if the camera was sitting at (0, 0, -z)?

Answer 1

I assume you're trying to figure out what the eye is looking at. I think I would do this:

calculate the vector from the camera to the eye using the dimensions of the glasses.

Make a guess about the geometric center within the interior the eye (your points are on the surface of the eye, not the center).

Then calculate the vector from the center of the eye to the point to get the 'looking at vector'

Does that help at all?

Answer 2

This is clearly a Photogrammetry problem which is not simple. Photogrammetry is taught in one, two or three classes (1-3 semesters) in technical universities.

At first, a straight line in 3dimensional space is projected as a (2dimensional) straight line in the image shot by the camera (and this irrelevant to the orientation of the camera). If the 3D line is indeed straight, and the 2D is not, then the camera is not "metric", which means that it distorts the image. Luckily, the main causes of the distortion can be compensated for by the Direct Linear Transform method (DLT), in most still cameras. The same should be true for video cameras, though I have not verified it.

Second, if you have the x, y coordinates of a point on the image (2 knowns), it is impossible to compute the X,Y,Z space coordinates of the same point (3 knowns). In order to do that, you must have a second image, of different orientation, where the same point has x', y' coordinates (2 knowns more). Then you are able to compute the X,Y,Z coordinates - in theory. In practice the computation is difficult because it involves non-linear Least Square Method (LSM).
Instead of a second Image, you may know that the points of interest lie on a surface (perhaps the surface of the eye). In this case the points should satisfy the equation of the surface which is of the form f(X, Y, Z)=0. With this equation combined with x, y coordinates of one image, you are able to compute the X, Y, Z coordinates. Again, the computations are not simple.

Third, the DLT equations are defined as:

     L1 X + L2 Y + L3 Z + L4
x = --------------------------
     L9 X + L10 Y + L11 Z + 1

     L5 X + L6 Y + L7 Z + L8
y = --------------------------
     L9 X + L10 Y + L11 Z + 1

The Li coefficients are unknown and can be computed, if you have at least 6 control points. A control point is a point with known X, Y, Z (3D) coordinates and known x, y (2D) coordinates on the image.

Once you compute Li, you do the same with a second image, whose orientation is different to the first image. Thus:

      L1' X + L2' Y + L3' Z + L4'
x' = -----------------------------
      L9' X + L10' Y + L11' Z + 1

      L5' X + L6' Y + L7' Z + L8'
y' = -----------------------------
      L9' X + L10' Y + L11' Z + 1

Fourth, now that you have the Li and the Li' coefficients, you are finally able to compute the X, Y, Z coordinates of an arbitrary point whose projection to the first image is x, y and the projection to the second image is x', y'. You solve the 4 DLT equations for X, Y, Z.

As a last thought, you can use control curves instead of control points, but this a field of active research.