Using Effect For Fog of War

Question

I'm trying to apply fog of war to areas on the screen not currently visible to the player. I do this by rendering the game content in one RenderTarget and the the fog of war into another, and then I merge them with an effect file that takes the color from the game RenderTarget and the alpha from the fog of war render target. The FOW RenderTarget is black where the FOW appears, and white where it doesn't.

This does work, but it colors the fog of war (the unrevealed locations) white instead of the intended color of black.

Before applying the effect I clear the backbuffer of the device to white. When I try to clear it to black, non of the fog of war appears at all, which I assume is a product of alpha blending with black. It works for all other colors, however - giving the resulting screen a tint of that color.

How do I archieve a black fog while still being able to do alpha blending between the two render targets?

The rendering code for applying the FOW:

private RenderTarget2D mainTarget;
private RenderTarget2D lightTarget;

   private void CombineRenderTargetsAndDraw()
    {
        batch.GraphicsDevice.SetRenderTarget(null);
        batch.GraphicsDevice.Clear(Color.White);

        fogOfWar.Parameters["LightsTexture"].SetValue(lightTarget);

        batch.Begin(SpriteSortMode.Immediate, BlendState.AlphaBlend);
        fogOfWar.CurrentTechnique.Passes[0].Apply();
        batch.Draw(
            mainTarget,
            new Rectangle(0, 0, batch.GraphicsDevice.PresentationParameters.BackBufferWidth, batch.GraphicsDevice.PresentationParameters.BackBufferHeight),
            Color.White
        );

        batch.End();
    }

The effect file I'm using to apply the FOW:

texture LightsTexture;

sampler  ColorSampler  : register(s0);

sampler LightsSampler = sampler_state{
    Texture = <LightsTexture>;
};

struct VertexShaderOutput
{
    float4 Position : POSITION0;
    float2 TexCoord : TEXCOORD0;
};

float4 PixelShaderFunction(VertexShaderOutput input) : COLOR0
{
    float2 tex = input.TexCoord;

    float4 color = tex2D(ColorSampler, tex);
    float4 alpha = tex2D(LightsSampler, tex);

    return float4(color.r, color.g, color.b, alpha.r);
}

technique Technique1
{
    pass Pass1
    {
        PixelShader = compile ps_2_0 PixelShaderFunction();
    }
}

Answer 1

It sounds like you are running into the situation that Shawn Hargreaves described in this blog post:
Premultiplied Alpha and Image Composition

Try switching to pre-multiplied alpha as he suggests and let us know the outcome :-)

Answer 2

I haven't looked too deeply into XNA and can't set it up at the moment to check with the CTP of 4.0, but from reading the documentation it looks like you're doubling up. From my understanding, BlendState.AlphaBlend causes the SpriteBatch to alpha blend each sprite that is drawn.

According to the documentation it uses the default settings of

ColorSourceBlend = Blend.One,
AlphaSourceBlend = Blend.One,

ColorDestinationBlend = Blend.InverseSourceAlpha,
AlphaDestinationBlend = Blend.InverseSourceAlpha,

Which should have the following impact:

One: Each component of the color is multiplied by (1, 1, 1, 1).

InverseSourceAlpha: Each component of the color is multiplied by the inverse of the alpha value of the source. This can be represented as (1 − As, 1 − As, 1 − As, 1 − As), where As is the alpha destination value.

So if you're using Blend.AlphaBlend you should be able to clear your back buffer to black and then only render your map sprites, directly lowering the alpha where it should be faded. This should simply fade the map pixels where 'covered' by FOW, causing it to alpha blend with the black of the back buffer.

If you want to take the shader approach you can alpha blend in the shader between two render targets. I'm a bit confused from your explanation on how exactly you actually want to blend. You say you want to alpha blend, but that your render target of FOW is black where there is fog and white where there isn't. If you're using alpha to control the blending, your fog colour should be uniform (or textured as appropriate) across the whole render target, as per what appearance you want. Say you want black fog, your colours should be (0, 0, 0) across the whole render target, and the alpha value should change depending on where you want to show it.

Using this approach you could change your shader to something like the following:

float4 PixelShaderFunction(VertexShaderOutput input) : COLOR0
{
    float2 tex = input.TexCoord;

    float4 color = tex2D(ColorSampler, tex);
    float4 fow = tex2D(LightsSampler, tex);

    // color * inverse of FOW alpha + FOW * FOW alpha
    return float4(color.r * (1-fow.a) + fow.r * fow.a,
                  color.g * (1-fow.a) + fow.g * fow.a,
                  color.b * (1-fow.a) + fow.b * fow.a,
                  1);
}

I haven't done HLSL shaders for a while but I'm sure that can be written much easier. Either way, if FOW is black (and back buffer is cleared to black, and you're not stacking various alpha-blended sprites on top of each other - see Joel's answer), that approach is no different to directly setting the alpha of the map sprite based on whether there is FOW or not.