Introduction

This is an article about learning the sparks of the spark class. I encountered a problem on the way. I simply recorded it. Although the problem has not been solved, it will be continuously updated.

正文

Recently I am learning spark, I feel that there is still a lot of resistance. I heard that the latest spark 2.3.0 can run directly on the kubernetes cluster, so I want to try it. Just in hand, there is google’s free trial of 300 US dollars, quickly create a k8s cluster! (I won’t be able to leave a message in the comment area)

Follow the official spark documentation: [https://spark.apache.org/docs/2.3.0/running-on-kubernetes.html] (https://spark.apache.org/docs/2.3.0/running-on -kubernetes.html)

Here is a brief summary:

  1. Download the spark file
    Download address: [http://spark.apache.org/downloads.html] (http://spark.apache.org/downloads.html)

  2. One line of commands starts

    Bin/spark-submit \
        --master k8s://https://107.167.187.109\
        --deploy-mode cluster \
        --name spark-pi \
        --class org.apache.spark.examples.SparkPi \
        --conf spark.executor.instances=3 \
        --conf spark.kubernetes.container.image=zhouzhipeng/my-spark \
            Local:///opt/spark/examples/jars/spark-examples_2.11-2.3.0.jar

Description
–master k8s://https://107.167.187.109 This address can be viewed using kubectl cluster-info.

I thought it would be very simple to run, but after the above command is finished, it has been stuck in the following place:

2018-05-28 14:38:56 INFO Client: 54 - Waiting for application spark-pi to finish...

Checked a lot of information has not been resolved, I do not know if it is because of the issue of the spark version, the guest officer, have the trouble to know how to solve help in the comment area to leave a message, thank you!

Last modified: 2019年3月29日

Author

Comments

Write a Reply or Comment

Your email address will not be published.