World-Coordinate Issues with gluUnProject()

Question

I'm currently calling Trace (method below) from a game loop. Right now all I'm trying to do is get the world coordinates from the screen mouse so I can move objects around in the world space. The values I'm getting from gluUnProject are however; puzzling me.

I was using glReadPixel(...) to get the Z value but that produced little to no movement in the object I was drawing and the resulting vector ended up being the same as my cameras location (except for the tiny decimal changes due to mouse movement), so I decided to get rid of the call and replace the Z value with 1.

My question is: Does the following code look right to you? Every example I've seen thusfar is either identical or -very- similar but I can't seem to produce correct results, even if I lock down the Y axis. If the code is correct, then I'm guessing that I'm just not using the resulting vector properly. Should I not be able to draw an object or point directly with the resulting vector or do I have to do something else with it, like normalize?

The current render mode is GL_RENDER and I am using glFrustum with a NearZ value of 1 and FarZ value of 2048, to create a perspective. There is also a series of viewports created along with scissors, with a size and width of 512x768 and positioned in each corner of a 1024x768 window. Trace(...) is called in between rendering of the upper left viewport and is the only perspective projection, while the other viewports are orthographic. FOV is set to 45.

void VideoWindow::Trace(int cursorX, int cursorY)
{
    double objX, objY, objZ;//holder for world coordinates
    GLint view[4];//viewport dimensions+pos
    GLdouble  p[16];//projection matrix
    GLdouble  m[16];//modelview matrix
    GLdouble z;//Z-Buffer Value?

    glGetDoublev (GL_MODELVIEW_MATRIX, m);
    glGetDoublev (GL_PROJECTION_MATRIX,p);
    glGetIntegerv( GL_VIEWPORT, view );

    //view[3]-cursorY = conversion from upper left (0,0) to lower left (0,0)
    //Unproject 2D Screen coordinates into wonderful world coordinates
    gluUnProject(cursorX, view[3]-cursorY, 1, m, p, view, &objX, &objY, &objZ);

    //Do something useful here???
}

Any ideas?

Edit: I've changed the winZ value to 0.5 instead of 1 which gives a vector thats more reasonable but drawing a point still wasn't matching the mouse. I found out that the value of view[3] was 384 which is correct for the viewport I'm using but I replaced it with 768 (the actual window size) and the point followed the mouse 100%. Further experimentation reveals that I can't use the coordinates to move around a 3D object in the perspective world space using this these coordinates however moving around 3D object in Orthographic space works fine.

Answer 1

just a thought, but if the third argument to gluUnProject is the z distance to the camera, wouldn't any point you draw at that location be on the near clipping plane of your frustum? Better make that z value a bit higher.

Answer 2

The winz argument to gluUnproject specifies the depth from the camera at which you're "picking" your points. As you've stated this coordinate should be in the [0, 1] range.

Some tutorials like NeHes read out the z coordinate from the depth buffer so that you "pick" at the right depth, of course for this to work you'll have to do the gluUnproject after you've rendered everything else.

Regardless, if you set winz to 0.5 or something (not 0 or 1 or the point will end up on the near or far clip plane, and maybe culled) and do the following:

gluUnProject(cursorX, view[3]-cursorY, 0.5, m, p, view, &objX, &objY, &objZ);
//Do something useful here???
glPointSize(10);
glBegin(GL_POINTS);
glColor3f(1, 0, 0);
glVertex3f(objX, objY, objZ);
glEnd();

You should end up with a red blob at the mouse pointer (provided nothing else overdraws it afterwards and you don't have any funny render states which renders the point invisible).