You might want to find the convex hull of all the points in your polygon.
One algorithm for doing this is Graham-Scan with complexity O(nlgn). From Cormen:
Let Q be the set of all points in your polygon
Graham-Scan(Q)
1 let p0 be the point in q with the minimum y-coordinate or the leftmost in case of tie
2 let (p1, p2,...,pm) be the remaining points in Q, sorted by polar angle around p0
if more than one point shares the same polar angle, keep the farthest point
3 let S be an empty stack
4 PUSH(p0, S)
5 PUSH(p1, S)
6 PUSH(p2, S)
7 for i = 3 to m
8 while the angle formed by points NEXT_TO_TOP(S), TOP(S), and pi makes a non-left turn
9 POP(S)
10 PUSH(pi, S)
11 return S
S now contains all of the outer points of your polygon. You'll have to do some polar math, but that's pretty simple. To sort by polar order sort all points on their cotangent with your bottom most point. I forget the code for checking a right turn, but it's on the internet if you just search fro Graham-Scan. Hope this helps!