Preqrequisites

There are a few requirements to achieve items mentioned in previous chapter. We submit spark jobs to YARN resource manager via a json input, which needs some information about

• artefacts present in hadoop

• Some environment information within hadoop cluster

• All nodes in the cluster should have java 1.8

• For running python libraries via TDSS all nodes in cluster should have python 2.7

• We use python egg packages like scikit_learn-0.19.1-py2.7-linux-x86_64.egg. So it will be good if every node of cluster is based on ideally RHEL 7

• Assuming when we get access to mutlitenant hdp cluster, we will be assigned a specific "QUEUE" for requesting YARN based resources. We need to know the name of that queue.

• It is good to use right spark-assembly with correct hadoop version. For example if we are using hadoop version 2.6 for setting up hadoop cluster, it is good to use a spark assembly which is built with hadoop-2.6 libraries. For example spark-assembly-1.6.1-hadoop2.6.0.jar

There are two types of spark jobs we submit to yarn

1. Pure java / scala based: For Spark Jobs that runs pure scala code, our JSON input requires only one host information. JAVA_HOME

   {

   "key": "**JAVA\_HOME**",
   
   "value": "**/usr/lib/jvm/java-1.8.0-openjdk**"

   }

2. PySpark Python based: For python based jobs, all the nodes of the hadoop cluster should have python2.7. If the default python on each node is pointing to python2.7, then we are good to go.

Sample Json Request Made to YARN Resource Manager for submitting a spark job.

{
    "am-container-spec": {
        "commands": {
            "command": "SOD_ENV=custom /usr/lib/jvm/java-1.8.0-openjdk/bin/java .... org.apache.spark.deploy.SparkSubmit --verbose  --queue <QUEUE_NAME>  .... "
        },
        "environment": {
            "entry": [{
                    "key": "JAVA_HOME",
                    "value": "/usr/lib/jvm/java-1.8.0-openjdk"
                },
                {
                    "key": "SPARK_YARN_MODE",
                    "value": true
                },
                {
                    "key": "HDP_VERSION",
                    "value": "2.6.0"
                },
                {
                    "key": "CLASSPATH",
                    "value": "{{PWD}}<CPS>__spark__.jar<CPS>{{PWD}}/__app__.jar<CPS>....."
                },
                {
                    "key": "SPARK_YARN_CACHE_FILES",
                    "value": "hdfs://11.0.0.226:54310/scb/tdss/lib/transformation-assembly-0.1.1.jar#__app__.jar......."
                },
                {
                    "key": "SPARK_YARN_CACHE_FILES_FILE_SIZES",
                    "value": "118507918,187698038...."
                },
                {
                    "key": "SPARK_YARN_CACHE_FILES_TIME_STAMPS",
                    "value": "1516363734946,1516017644790...."
                },
                {
                    "key": "SPARK_YARN_CACHE_FILES_VISIBILITIES",
                },
                {
                    "key": "PYTHONPATH",
                    "value": "{{PWD}}/__pyspark__.zip<CPS>{{PWD}}/__py4jsrc__.zip<CPS>{{PWD}}/__pandas__.egg<CPS>{{PWD}}/__numpy__.egg<CPS>"
                },
                {
                    "key": "PYTHON_EGG_CACHE",
                    "value": "{{PWD}}/"
                },
                {
                    "key": "SPARK_HOME",
                    "value": "/dev/null"
                }
            ]
        },
        "local-resources": {
            "entry": [{
                    "key": "__spark__.jar",
                    "value": {
                        "resource": "hdfs://11.0.0.226:54310/scb/tdss/spark/spark-assembly-1.6.1-hadoop2.6.0.jar",
                        "size": "187698038",
                        "timestamp": "1516017644790",
                        "type": "FILE",
                        "visibility": "APPLICATION"
                    }
                },
                {
                    "key": "__app__.jar",
                    "value": {
                        "resource": "hdfs://11.0.0.226:54310/scb/tdss/lib/transformation-assembly-0.1.1.jar",
                        "size": "118507918",
                        "timestamp": "1516363734946",
                        "type": "FILE",
                        "visibility": "APPLICATION"
                    }
                },
                ........
                ........
                ........                      
                {
                    "key": "__wrapper__.py",
                    "value": {
                        "resource": "hdfs://11.0.0.226:54310/scb/tdss/udf/driver_wrapper.py",
                        "size": "13212",
                        "timestamp": "1516448083899",
                        "type": "FILE",
                        "visibility": "APPLICATION"
                    }
                }
            ]
        }
    },
    "application-id": "application_151636000046108_00014",
    "application-name": "TRANSFORMATION_JOB_42",
    "application-type": "YARN",
    "queue": "QUEUE_NAME",
    "keep-containers-across-application-attempts": false,
    "max-app-attempts": 1,
    "resource": {
        "memory": "2500",
        "vCores": "1"
    },
    "unmanaged-AM": false
}