This is an article about learning the sparks of the spark class. I encountered a problem on the way. I simply recorded it. Although the problem has not been solved, it will be continuously updated.
Recently I am learning spark, I feel that there is still a lot of resistance. I heard that the latest spark 2.3.0 can run directly on the kubernetes cluster, so I want to try it. Just in hand, there is google’s free trial of 300 US dollars, quickly create a k8s cluster! (I won’t be able to leave a message in the comment area)
Follow the official spark documentation: [https://spark.apache.org/docs/2.3.0/running-on-kubernetes.html] (https://spark.apache.org/docs/2.3.0/running-on -kubernetes.html)
Here is a brief summary:
One line of commands starts
Bin/spark-submit \ --master k8s://https://18.104.22.168\ --deploy-mode cluster \ --name spark-pi \ --class org.apache.spark.examples.SparkPi \ --conf spark.executor.instances=3 \ --conf spark.kubernetes.container.image=zhouzhipeng/my-spark \ Local:///opt/spark/examples/jars/spark-examples_2.11-2.3.0.jar
–master k8s://https://22.214.171.124 This address can be viewed using
I thought it would be very simple to run, but after the above command is finished, it has been stuck in the following place:
2018-05-28 14:38:56 INFO Client: 54 - Waiting for application spark-pi to finish...
Checked a lot of information has not been resolved, I do not know if it is because of the issue of the spark version, the guest officer, have the trouble to know how to solve help in the comment area to leave a message, thank you!