I think the answer you want is "no", but I also think you might want to consider your question more carefully.
The problem is that you are using continuous coordinates, so in principle you can't use a linear sort. (In practice, you can use use, say, a radix sort to handle a fixed-size coordinate, but in practice this may actually be slower than a standard O(N log N) sort, due to the overhead involved...)
The theoretical rule is: whenever you have a situation where you can only compare your values, a general sort cannot be faster than O(N log N).
You mentioned an unspecified property that "might be exploited most of the time". The problem is that O() notation is an asymptotic, worst-case, theoretical property, so "most of the time" won't make a difference. A specific property of the input can often be exploited in this way, but:
- the O() speedup is very sensitive to exactly what your property is
- an algorithm with a better O() might be much slower for your practical application
- the most effective technique to practically exploit your input is often very different from the technique with the best asymptotic properties
Unfortunately, when tweaking your algorithm to exploit your input, it is easy to miss an obscure worst-case scenario which will kill your performance in unexpected ways, long after you have released it. The most important thing about the O() of your algorithm is to stop it from blowing up badly like this.
Note that, for this purpose, O(N log N) is near-linear, and using a standard, well-behaved library sort may well be the right choice.