You don't want to double count any segment, so your algorithm should be a double for loop. The outer loop goes from A to M (you don't need to check N, because there'll be nothing left for it to connect to), each time looping from curPoint to N, calculating each distance. You add all the distances, and divide by the number of points (n-1)^2/2. Should be pretty simple.
There aren't any standard algorithms for improving on this that I'm aware of, and this isn't a widely studied problem. I'd guess that you could get a pretty reasonable estimate (if an estimate is useful) by sampling distances from each point to a handful of others. But that's a guess.
(After seeing your code example) Here's another try:
public double avgDistanceInCluster() {
double totDistance = 0.0;
for (int i = 0; i < bigCluster.length - 1; i++) {
for (int j = i+1; j < bigCluster.length; j++) {
totDistance += distance(bigCluster[i], bigCluster[j]);
}
}
return totDistance / (bigCluster.length * (bigCluster.length - 1)) / 2;
}
Notice that the limit for the first loop is different.
Distance between two points is probably sqrt((x1 - x2)^2 + (y1 -y2)^2)
.