-
Notifications
You must be signed in to change notification settings - Fork 15
Description
Hi ,
i used custom script in ARM Template while creating HDInsight Spark Cluster. The template was getting executed successfully , but the script was not getting executed. I used some environment variables in my script which was not getting resolved. I spent a lot of time to figure it out the issue.
"scriptActions": [{
"name": "createLivyFolder",
"uri": "[parameters('scriptActionUri')]"
}]
The below challenges i faced
- Not able to get the path of the folder from where the script was executed in HDInsight Cluster.
- Not able to get the path of error log or Stack-trace of script action.
After spending a lot of time i got the log from the Azure blob storage and was able to get the cause of the problem.
Can you please add to the doc how to debug the script action in case of failure or get the stack trace in case of success/failure.
Please let us know if we want to use some environment variables in script action like $SPARK_HOME,
How to do that. The below statement was failing in script actions.
sudo cp postgresql-9.4.1210.jre7.jar $SPARK_HOME/jars/