Opengl Selection with Alpha Test

Question

From what I've been seeing around the internet, this problem can't really be solved with my approach.

I am currently writing a program that uses a selection buffer pass over all the objects in the scene. However, one of the objects is a texture in which a large part of it has alpha 0. It works fine when rendering, the alpha values are not displayed, but when in selection mode, the alpha test is skipped, and hovering the mouse over the transparent areas selects the object instead of whatever is behind it.

Is there any way to enable alpha testing in selection mode?

Answer 1

Looks like glReadPixels is what you are looking for

http://www.opengl.org/sdk/docs/man/xhtml/glReadPixels.xml

Use something like:

glReadPixels (0, 0, XSCREEN, YSCREEN, GL_ALPHA, GL_UNSIGNED_BYTE, buffer);

Answer 2

I think what you mean is the alpha blending is skipped. I haven't tested it, but it appears from the comment here, that alpha tests are still used in GL_SELECT mode. So you would want to glEnable(GL_ALPHA_TEST) and then use glAlphaFunc() to set your threshold for how much alpha constitutes a hit. To test before you just blindly try selecting these objects, just render it. You'll see that OpenGL only renders part of your polygon; and any of that part would constitute a click, while any of the not-drawn part won't be hit with your GL_SELECT.

Please comment and let me know how it works out. If you still can't get it to work, I can take the time to write a test case for you, but I'd rather not spend the time if you can get it to work on your own. :)

Also, for reference, the OpenGL docs for Alpha test and the manpage for glAlphaFunc.