How to use OpenCV cvProjectPoints2 function

Question

I'm having some troubles with cvProjectPoints2 function. Following is the function overview from O'Reilly's "Learning OpenCV" book:

void cvProjectPoints2(
const CvMat* object_points,
const CvMat* rotation_vector,
const CvMat* translation_vector,
const CvMat* intrinsic_matrix,
const CvMat* distortion_coeffs,
CvMat* image_points,
);

The first argument, object_points, is the list of points you want projected; it is just an N-by-3 matrix containing the point locations. You can give these in the object’s own local coordinate system and then provide the 3-by-1 matrices rotation_vector* and translation_vector to relate the two coordinates. If in your particular context it is easier to work directly in the camera coordinates, then you can just give object_points in that system and set both rotation_vector and translation_vector to contain 0s.†

The intrinsic_matrix and distortion_coeffs are just the camera intrinsic information and the distortion coefficients that come from cvCalibrateCamera2() discussed in Chapter 11. The image_points argument is an N-by-2 matrix into which the results of the computation will be written.

First of all, there seems to be a bug with object_points array. If there is only one point, that is N=1, the program crashes. Anyway, I have several camera intrinsic parameters and projection matrices. The distortion coefficients are given as 0, that is, there is no distortion. For simplicity, assume I have 2 cameras:

double intrinsic[2][3][3] = {
//camera 0
1884.190000,    0, 513.700000,
0.0,    1887.490000, 395.609000,
0.0,    0.0, 1.0,
//camera 4
1877.360000,    0.415492, 579.467000,
0.0,    1882.430000, 409.612000,
0.0,    0.0, 1.0
};

double projection[2][3][4] = {
//camera 0
0.962107,   -0.005824, 0.272486, -14.832727,
0.004023,   0.999964, 0.007166, 0.093097,
-0.272519,  -0.005795, 0.962095, -0.005195,
//camera 4
1.000000,   0.000000, -0.000000, 0.000006,
0.000000,   1.000000, -0.000000, 0.000001,
-0.000000,  -0.000000, 1.000000, -0.000003
};

As far as I understand, this information is sufficient to project any point (x,y,z) on any camera view. Here, in x,y,z coordinates, the optical center of camera 4 is the origin of world coordinates.

Here's my code:

#include <cv.h>
#include <highgui.h>
#include <cvaux.h>
#include <cxcore.h>
#include <stdio.h>

double intrinsic[2][3][3] = {
//0
1884.190000,    0, 513.700000,
0.0,    1887.490000, 395.609000,
0.0,    0.0, 1.0,
//4
1877.360000,    0.415492, 579.467000,
0.0,    1882.430000, 409.612000,
0.0,    0.0, 1.0
};

double projection[2][3][4] = {
//0
0.962107,   -0.005824, 0.272486, -14.832727,
0.004023,   0.999964, 0.007166, 0.093097,
-0.272519,  -0.005795, 0.962095, -0.005195,
//4
1.000000,   0.000000, -0.000000, 0.000006,
0.000000,   1.000000, -0.000000, 0.000001,
-0.000000,  -0.000000, 1.000000, -0.000003
};


int main() {
    CvMat* camera_matrix[2]; //
    CvMat* rotation_matrix[2]; //
    CvMat* dist_coeffs[2]; 
    CvMat* translation[2];
    IplImage* image[2];
    image[0] = cvLoadImage("color-cam0-f000.bmp", 1);
    image[1] = cvLoadImage("color-cam4-f000.bmp", 1);
    CvSize image_size;
    image_size = cvSize(image[0]->width, image[0]->height);

    for (int m=0; m<2; m++) {
     camera_matrix[m] = cvCreateMat(3, 3, CV_32F);
     dist_coeffs[m] = cvCreateMat(1, 4, CV_32F);
     rotation_matrix[m] = cvCreateMat(3, 3, CV_32F);
     translation[m] = cvCreateMat(3, 1, CV_32F);
    }

    for (int m=0; m<2; m++) {
     for (int i=0; i<3; i++)
      for (int j=0; j<3; j++) {
       cvmSet(camera_matrix[m],i,j, intrinsic[m][i][j]);
       cvmSet(rotation_matrix[m],i,j, projection[m][i][j]);
      }
     for (int i=0; i<4; i++)
      cvmSet(dist_coeffs[m], 0, i, 0);
     for (int i=0; i<3; i++)
      cvmSet(translation[m], i, 0, projection[m][i][3]);
    }

    CvMat* vector = cvCreateMat(3, 1, CV_32F);
    CvMat* object_points = cvCreateMat(10, 3, CV_32F);
    cvmSet(object_points, 0, 0, 1000);
    cvmSet(object_points, 0, 1, 500);
    cvmSet(object_points, 0, 2, 100);

    CvMat* image_points = cvCreateMat(10, 2, CV_32F);
    int m = 0;
    cvRodrigues2(rotation_matrix[m], vector);
    cvProjectPoints2(object_points, vector, translation[m], camera_matrix[m], dist_coeffs[m], image_points);
    printf("%f\n", cvmGet(image_points, 0, 0));
    printf("%f\n", cvmGet(image_points, 0, 1));
    return 0;
}

The images are 1024*768, and the visible part of z is know to be between 44 and 120. So, the point should be seen on both of the cameras, right? But the result is absolutely wrong. Even for m = 1. What am I doing wrong?

Answer 1

I haven't had much time to look at your code, but what is the 0.415492 factor in the camera intrinsics matrix for camera 4? I expected that to be 0.0.

Answer 2

Yeah, cvProjectPoints is used to project array of points. You can project one point with simple matrix operations:

CvMat *pt = cvCreateMat(3, 1, CV_32FC1);
CvMat *pt_rt = cvCreateMat(3, 1, CV_32FC1);
CvMat *proj_pt = cvCreateMat(3, 1, CV_32FC1);
cvMatMulAdd(rotMat, pt, translation, pt_rt);
cvMatMul(intrinsic, pt_rt, proj_pt);
// convertPointsHomogenious might be used
float scale = (float)CV_MAT_ELEM(*proj_pt, float, 2, 0); 
float x = CV_MAT_ELEM(*proj_pt, float, 0, 0) / scale;
float y = CV_MAT_ELEM(*proj_pt, float, 1, 0) / scale;
CvPoint2D32f img_pt = cvPoint2D32f(x, y);