I'm not sure if this is possible to do, but it's worth a shot. I'm using the stencil buffer to reduce overdraw for light volumes in a deferred renderer using this algorithm (when the camera is outside the volume):
- Using a cheap shader, draw back faces with depth testing set to LEQUAL, marking them in the stencil buffer.
- Using the expensive lighting shader, draw front faces with depth testing set to GEQUAL.
This will cause only pixels within the light volume to be shaded. The problem with this comes when drawing multiple lights. First, since state changes are expensive, it's probably not the best idea to repeatedly switch between the cheap and expensive shader for each light. Ideally I'd like to take advantage of all 8 bits of the stencil buffer by rendering 8 light volumes with the cheap shader followed by 8 light volumes with the expensive shader. However, issues arise when lights overlap, as there's no way to tell which pixels belong to which lights.
The solution that comes to mind would be to use 1 bit in the stencil buffer per light. So, for light n, mark the nth bit in the stencil buffer in the cheap pass, then only render pixels with that bit on during the expensive pass.
I haven't used the stencil buffer before, but from what I'm reading this doesn't seem possible. For this to work I'd have to set the stencil buffer using bitwise OR and the stencil function would have to be bitwise AND. However, the only operations on the stencil buffer I can see are: KEEP, ZERO, REPLACE, INCR, DECR, and INVERT, and the only functions are: NEVER, ALWAYS, LESS, EQUAL, LEQUAL, GEQUAL, GREATER, and NOTEQUAL.
Is there any way to somehow get this OR and ANDing behavior using the stencil buffer? And if not, is there an alternative approach to efficiently rendering light volumes?