Given normal map in world space what is a suitable algorithm to find edges?

Question

If I have the vertex normals of a normal scene showing up as colours in a texture in world space is there a way to calculate edges efficiently or is it mathematically impossible? I know it's possible to calculate edges if you have the normals in view space but I'm not sure if it is possible to do so if you have the normals in world space (I've been trying to figure out a way for the past hour..)

I'm using DirectX with HLSL.

Answer 1

It will depend on how many colors your image contain, and how they merge: sharp edges, dithered, blended,...
Since you say you have the vertex normals I am assuming that you can access the color-information on a single plane.
I have used two techniques with varying success:

1. I searched the image for local areas of the same color (RGB) and then used the highest of R, G or B to find the 'edge' - that is where the selected R,G or B is no longer the highest value;
2. the second method I used is to reduce the image to 16 colors internally, and it is easy to find the outlines in this case.

To construct vectors would then depend on how fine you want the granularity of your 'wireframe'-image to be.

Answer 2

if ( normalA dot normalB > cos( maxAngleDiff )

then you have an edge. It won't be perfect but it will definitely find edges that other methods won't.

Or am i misunderstanding the problem?

Edit: how about, simply, high pass filtering the image?

Answer 3

I assume you are trying to make cartoon style edges for a cell shader?

If so, simply make a dot product of the world space normal with the world space pixel position minus camera position. As long as your operands are all in the same space you should be ok.

float edgy = dot(world_space_normal, pixel_world_pos - camera_world_pos);

If edgy is near 0, it's an edge.

If you want a screen space sized edge you will need to render additional object id information on another surface and post process the differences to the color surface.