This loop does basically the same thing, in a more Javascript-y way:
for (var div = 1, radix = 16; div < 65536 * 65536; div *= radix) {
var piles = [];
for (var i = 0; i < a.length; ++i) {
var p = Math.floor(a[i] / div) % radix;
(piles[p] || (piles[p] = [])).push(a[i]);
}
for (var i = 0, ai = 0; i < piles.length; ++i) {
if (!piles[i]) continue;
for (var pi = 0; pi < piles[i].length; ++pi)
a[ai++] = piles[i][pi];
}
}
Instead of doing it like a C programmer might, this loop builds a list of lists, one list for each possible 4-bit value. I avoid bit-shift operators because this is Javascript and while they do work, things get funny when numbers get large.
Starting with the low 4 bits of each value in "a", the code copies that element of "a" to the end of one of the "piles", that being the one corresponding to the 4-bit value. It then gathers up the piles and rebuilds "a" starting from all of the values whose low 4 bits were 0, then 1, etc. (Clearly there'll be some gaps, so those are skipped.) At the end of each iteration of the overall loop, the divisor is multiplied by the radix, so that the next set of 4 bits will be examined.
Once the divisor has exhausted the available range of integers, it's done.
Note that this will only work for positive numbers. Doing this with negative numbers gets a little weird; it might be easier to strip out the negative numbers into a separate array, flip the sign, sort, then reverse. Sort the positive numbers, and then finally glue the reversed negative numbers (flipping the signs again) to the front of the sorted positive numbers.