A convex-hull is, by definition, 2D. Perhaps you mean a convex polyhedron?
Let your structure of your algorithm be thus:
int pixelMap[][] = zeroes(NUM_HORIZ_PIXELS,NUM_VERT_PIXELS);
for(int xpxl = 0; xpxl < NUM_HORIZ_PIXELS; xpxl++) {
for(int ypxl = 0; ypxl < NUM_VERT_PIXELS; ypxl++) {
float xcoord = getXCoordForPixel(xpxl);
float ycoord = getYCoordForPixel(ypxl);
if(withinHull(xcoord,ycoord)) {
pixelMap[xpxl][ypxl] = 1;
} // end if
} // end for y
} // end for x
Now you need to define your getCoord
functions (which are just converting from screen-space to real-space, and are pretty straightforward) and your withinHull
function.
Because you're dealing with 2d, it's pretty much safe to assume that your hull will end up being 2d - even if it's 3d, you'll wind up with it's 2d projection on the plane you're dealing with anyway. For algorithms for creating a hull from a set of points, see http://en.wikipedia.org/wiki/Convex_hull_algorithms (I remember learning the D&C algorithm in uni).
Once you have your 2d hull defined, you should be able to treat it as a polygon. From here, you can use a polygon intersection test to determine if your coordinates passed fall within the polygon. An easy one is to pick a point known to be outside the polygon, make a line between the coords given and your known outside point, and test intersection against each line of the polygon. Even number of intersections means point is not in the polygon, odd number means it is in the polygon. This algorithm also works for non convex polygons too, in case you ever need it again.
Matlab may or may not have some of these functions predefined (line intersection, polygon representation, etc.)
cheers!