How to load Java property files and use them in spark?

I want to store spark parameters (such as input file and output file) in the Java properties file and pass the file to spark driver I'm submitting a job using spark submit, but I can't find the parameters to pass the properties file Do you have any suggestions

Solution

Here I find a solution:

Profiles file: (myprofile. CONF) / / Note: type your key with "spark" Otherwise, the prop will be ignored

spark.myapp.input /input/path
spark.myapp.output /output/path

launch

$SPARK_HOME/bin/spark-submit --properties-file  mypropsfile.conf

How to call code: (the code inside)

sc.getConf.get("spark.driver.host")  // localhost
sc.getConf.get("spark.myapp.input")       // /input/path
sc.getConf.get("spark.myapp.output")      // /output/path
The content of this article comes from the network collection of netizens. It is used as a learning reference. The copyright belongs to the original author.
THE END
分享
二维码
< <上一篇
下一篇>>