I second groovingandi's suggestion of trying all polygons; you just have to be sure to check the validity of the polygon (no self-intersections, etc).
Now, if you want to work with lots of points... As MatlabDoug pointed out, the convex hull is a good place to start. Notice that the convex hull gives a polygon whose area is the maximum possible. The problem, of course, is that there could be points in the interior of the hull that are not part of the polygon. I propose the following greedy algorithm, but I am not sure if it guarantees THE maximum area polygon.
The basic idea is to start with the convex hull as a candidate final polygon, and carve out triangles corresponding to the unused points until all the points belong to the final polygon. At each stage, the smallest possible triangle is removed.
Given: Points P = {p1, ... pN}, convex hull H = {h1, ..., hM}
where each h is a point that lies on the convex hull.
H is a subset of P, and it is also ordered such that adjacent
points in the list of H are edges of the convex hull, and the
first and last points form an edge.
Let Q = H
while(Q.size < P.size)
% For each point, compute minimum area triangle
T = empty heap of triangles with value of their area
For each P not in Q
For each edge E of Q
If triangle formed by P and E does not contain any other point
Add triangle(P,E) with value area(triangle(P,E))
% Modify the current polygon Q to carve out the triangle
Let t=(P,E) be the element of T with minimum area
Find the ordered pair of points that form the edge E within Q
(denote them Pa and Pb)
Replace the pair (Pa,Pb) with (Pa,E,Pb)
Now, in practice you don't need a heap for T
, just append the data to four lists: one for P, one for Pa, one for Pb, and one for the area. To test if a point lies within a triangle, you only need to test each point against the lines forming the sides of the triangle, and you only need to test points not already in Q. Finally, to compute the area of the final polygon, you can triangulate it (like with the delaunay
function, and sum up the areas of each triangle in the triangulation), or you can find the area of the convex hull, and subtract out the areas of the triangles as you carve them out.
Again, I don't know if this greedy algorithm is guaranteed to find the maximum area polygon, but I think it should work most of the time, and is interesting nonetheless.