Hue使用Oozie执行Spark2任务

Oozie共享库添加Spark2

1. Oozie用户在/user/oozie/share/lib/lib_2019102914381创建spark2目录

    sudo -u oozie hdfs dfs -mkdir /user/oozie/share/lib/lib_20180605143536/spark2
   

   

2. 复制jar包到spark2

    cd /usr/hdp/3.1.0.0-78/spark2/jars
    sudo -u oozie hdfs dfs -put *.jar /user/oozie/share/lib/lib_2019102914381/spark2
   
    cd /usr/hdp/3.1.0.0-78/oozie/share/lib/spark
    sudo -u oozie hdfs dfs -put oozie-sharelib-spark-4.3.1.3.1.0.0-78.jar /user/oozie/share/lib/lib_2019102914381/spark2
   

3. 修改目录权限

    sudo -u hdfs hdfs dfs -chmod -R 775 /user/oozie/share/lib/lib_20180605143536/spark2
   

4. 更新Oozie的share-lib

    su oozie
    oozie admin -oozie http://10.47.123.40:11000/oozie -sharelibupdate
   

5. 查看Oozie的share-lib

    oozie admin -oozie http://10.47.123.40:11000/oozie -shareliblist
   

6. 创建spark任务

   

   点击Files，选择/user/oozie/share/lib/lib_2019102914381/spark2，点击选择此文件夹

   Jar/py name填写spark-examples_2.11-2.3.2.3.1.0.0-78.jar

   

   填写Main class：org.apache.spark.example.SparkPi

   

   Spark Master: yarn

   Mode: client

   oozie.action.sharelib.for.spark: spark2

   

   保存并执行