HLSL tex2d sampler seemingly using inconsistent rounding; why?

Question

Hello all,

I have code that needs to render regions of my object differently depending on their location. I am trying to use a colour map to define these regions.

The problem is when I sample from my colour map, I get collisions. Ie, two regions with different colours in the colourmap get the same value returned from the sampler.

I've tried various formats of my colour map. I set the colours for each region to be "5" apart in each case;

• Indexed colour
• RGB, RGBA: region 1 will have RGB 5%,5%,5%. region 2 will have RGB 10%,10%,10% and so on.
• HSV Greyscale: region 1 will have HSV 0,0,5%. region 2 will have HSV 0,0,10% and so on.

(Values selected in The Gimp)

The tex2D sampler returns a value [0..1].

[ I then intend to derive an int array index from region. Code to do with that is unrelated, so has been removed from the question ]

float region = tex2D(gColourmapSampler,In.UV).x;

Sampling the "5%" colour gave a "region" of 0.05098 in hlsl. From this I assume the 5% represents 5/100*255, or 12.75, which is rounded to 13 when stored in the texture. (Reasoning: 0.05098 * 255 ~= 13)

By this logic, the 50% should be stored as 127.5.
Sampled, I get 0.50196 which implies it was stored as 128.

the 70% should be stored as 178.5.
Sampled, I get 0.698039, which implies it was stored as 178.

What rounding is going on here?
(127.5 becomes 128, 178.5 becomes 178 ?!)

Edit: OK, http://en.wikipedia.org/wiki/Bankers_rounding#Round_half_to_even Apparently this is "banker's rounding". I have no idea why this is being used, but it solves my problem. Apparently, it's a Gimp issue.

---

I am using Shader Model 2 and FX Composer. This is my sampler declaration;

//Colour map
texture gColourmapTexture <
    string ResourceName = "Globe_Colourmap_Regions_Greyscale.png";
    string ResourceType = "2D";
>;
sampler2D gColourmapSampler : register(s1) = sampler_state {
Texture = <gColourmapTexture>;
#if DIRECT3D_VERSION >= 0xa00
    Filter = MIN_MAG_MIP_LINEAR;
#else /* DIRECT3D_VERSION < 0xa00 */
    MinFilter = Linear;
    MipFilter = Linear;
    MagFilter = Linear;
#endif /* DIRECT3D_VERSION */
    AddressU = Clamp; 
    AddressV = Clamp;
};   

Answer 1

I never used HLSL, but I did use GLSL a while back (and I must admit it's terribly far in my head).

One issue I had with textures is that 0 is not the first pixel. 1 is not the second one. 0 is the edge of the texture and 1 is the right edge of the first pixel. The values get interpolated automatically and that can cause serious trouble if what you need is precision like when applying a lookup table rather than applying a normal texture. You need to aim for the middle of the pixel, so asking for [0.5,0.5], [1.5,0.5] rather than [0,0], [1, 0] and so on.

At least, that's the way it was in GLSL.

Answer 2

Beware: region in levels[region] is rounded down. When you see 5 % in your image editor, the actual value in the texture 8b representation is 5/100*255 = 12.75, which may be either 12 or 13. If it is 12, the rounding down will hit you. If you want rounding to nearest, you need to change this to levels[region+0.5].

Another similar thing (already written by Louis-Philippe) which might hit you is texture coordinates rounding rules. You always need to hit a spot in the texel so that you are not in between of two texels, otherwise the result is ill-defined (you may get any of two randomly) and some of your source texels may disapper while other duplicate. Those rules are different for bilinar and point sampling, you may need to add half of texel size when sampling to compensate for this.

Answer 3

GIMP uses banker's rounding. Apparently.

This threw out my code to derive region indicies.