Examples of simple stats calculation with hadoop

Question

I want to extend an existing clustering algorithm to cope with very large data sets and have redesigned it in such a way that it is now computable with partitions of data, which opens the door to parallel processing. I have been looking at Hadoop and Pig and I figured that a good practical place to start was to compute basic stats on my data, i.e. arithmetic mean and variance.

I've been googling for a while, but maybe I'm not using the right keywords and I haven't really found anything which is a good primer for doing this sort of calculation, so I thought I would ask here.

Can anyone point me to some good samples of how to calculate mean and variance using hadoop, and/or provide some sample code.

Thanks

Answer 1

Pig latin has an associated library of reusable code called PiggyBank that has numerous handy functions. Unfortunately it didn't have variance last time I checked, but maybe that has changed. If nothing else, it might provide examples to get you started on your own implementation.

I should note that variance is difficult to implement in a stable way over huge data sets, so take care!

Answer 2

You might double check and see if your clustering code can drop into Cascading. Its quite trivial to add new functions, do joins, etc with your existing java libraries.

http://www.cascading.org/

And if you are into Clojure, you might watch these github projects: http://github.com/clj-sys

They are layering new algorithms implemented in Clojure over Cascading (which in turn is layered over Hadoop MapReduce).