A few points, for clarity.
Unless you specifically wish these squares to be square (and hence to not fit exactly in this parallel and perpendicular layout with regards to the meridians), these are not exactly squares. This is particularly visible if the dimensions of the square are big.
The question speaks of a [perfect] sphere. Matters would be somewhat different if we were considering the Earth (or other planets) with its flattened poles.
Following is a "algorithm" that would fit the bill, I doubt it is optimal, but could offer a good basis. EDIT: see Tom10's suggestion to work with the plain 3D distance between the points rather than the corresponding great cirle distance (i.e. that of the cord rather than the arc), as this will greatly reduce the complexity of the formulas.
Problem layout: (A, B and Sq as defined in the OP's question)
A : a given point the the surface of the sphere
Sq : a given "square" from the grid
B : solution to problem : point located within Sq which has the shortest
distance to A.
C : point at the center of Sq
Tentative algorithm:
Using the formulas associated with [Great Circle][1], we can:
- find the equation of the circle that includes A and C
- find the distance between A and C. See the [formula here][2] (kindly lifted
from Tom10's reply).
- find the intersect of the Great Circle arc between these points, with the
arcs of parallel or meridian defining the Sq.
There should be only one such point, unless this finds a "corner" of Sq,
or -a rarer case- if the two points are on the same diameter (see
'antipodes' below).
Then comes the more algorithmic part of this procedure (so far formulas...):
- find, by dichotomy, the point on Sq's arc/seqment which is the closest from
point A. We're at B! QED.
Optimization:
It is probably possible make a good "guess" as to the location
of B, based on the relative position of A and C, hence cutting the number of
iterations for the binary search.
Also, if the distance A and C is past a certain threshold the intersection
of the cicles' arcs is probably a good enough estimate of B. Only when A
and C are relatively close will B be found a bit further on the median or
parallel arc in these cases, projection errors between A and C (or B) are
smaller and it may be ok to work with orthogonal coordinates and their
simpler formulas.
Another approach is to calculate the distance between A and each of the 4
corners of the square and to work the dichotomic search from two of these
points (not quite sure which; could be on the meridian or parallel...)
( * ) *Antipodes case*: When points A and C happen to be diametrically
opposite to one another, all great circle lines between A and C have the same
length, that of 1/2 the circonference of the sphere, which is the maximum any
two points on the surface of a sphere may be. In this case, the point B will
be the "square"'s corner that is the furthest from C.
I hope this helps...