Are you trying to find which 3D point are visible by a camera? If so, you might find some interesting informations on this website: http://www.lighthouse3d.com/opengl/viewfrustum/.
In the following image, we can see the view frustum and your selection frustum (in red). Applying frustum visibility checks to your selection frustum should the same algorithm as the one used for frustum culling.
If you want a quick and non optimized solution:
GLdouble model_view[16];
glGetDoublev(GL_MODELVIEW_MATRIX, model_view);
GLdouble projection[16];
glGetDoublev(GL_PROJECTION_MATRIX, projection);
GLint viewport[4];
glGetIntegerv(GL_VIEWPORT, viewport);
for(unsigned i=0; i<points.size(); ++i){
GLdouble winX, winY, winZ;
gluProject(points[i].x, points[i].y, points[i].z, model_view, projection, viewport, &winX, &winY, &winZ);
if(selectionMinX <= winX && winX <= selectionMaxX && selectionMinY <= winY && winY <= selectionMaxY && winZ>=0 && winZ<=1){
/// point is selected
}
}