Tuesday, April 23, 2013

How to Build Oozie with Different Versions of Hadoop

After downloading Oozie code with

svn checkout http://svn.apache.org/repos/asf/oozie/tags/release-3.3.0/ .

and then building it with Hadoop 1.1.0 with the familiar

mvn clean compile -Dhadoop.version=1.1.0

I got the following error:

[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 1:06.497s
[INFO] Finished at: Tue Apr 23 12:36:53 BRT 2013
[INFO] Final Memory: 20M/67M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal on project oozie-sharelib-distcp: Could not resolve dependencies for project org.apache.oozie:oozie-sharelib-distcp:jar:3.3.0: Could not find artifact org.apache.oozie:oozie-hadoop-distcp:jar:1.1.0.oozie-3.3.0 in central (http://repo1.maven.org/maven2) -> [Help 1]


Reading a bit about it, and checking some pom files, I realized that inside the hadoolibs directory (inside oozie home), there are three sub-directories with the hadoop version hard coded on their poms.
So when you pass the -Dhadoop.version, these pom don't "change"! And they continue on using their pre-defined version of Hadoop!

I talked to the community guys from Oozie, and they say that the recommended thing to do is to change the pom files itself, and not pass by parameter.

Resuming, if you want to build oozie 3.3 with a different Hadoop, edit these pom files:

oozie_home/hadooplibs/hadoop-1/pom.xml
oozie_home/hadooplibs/hadoop-distcp-1/pom.xml
oozie_home/hadooplibs/hadoop-test-1/pom.xml
oozie_home/pom.xml

Setting the desired version of Hadoop. This off courseif you are building against Hadoop 1.x. If you are building oozie with Hadoop 2.x, edit:


oozie_home/hadooplibs/hadoop-2/pom.xml
oozie_home/hadooplibs/hadoop-distcp-2/pom.xml
oozie_home/hadooplibs/hadoop-test-2/pom.xml
oozie_home/pom.xml


4 comments: