How do I use a more recent version of a hadoop/lib jar in my map/reduce jobs?

Question

Hadoop currently ships with commons-httpclient-3.0.1.jar in its lib folder.

If I have a map/reduce task that requires commons-httpclient-3.1.jar, it does not seem to be sufficient to bundle this jar in the lib folder of my hadoop jar (as one would do with any normal external jar dependencies), as hadoop seems to be loading the previous jar from its lib folder and not using the one I have supplied.

It is not possible to use different versions of jars which hadoop also uses?

Answer 1

If all else fails, use the maven-shade-plugin to rename the package of commons-httpclient so that you have yours and hadoop has its.