How to load Java property files and use them in spark?
•
Java
I want to store spark parameters (such as input file and output file) in the Java properties file and pass the file to spark driver I'm submitting a job using spark submit, but I can't find the parameters to pass the properties file Do you have any suggestions
Solution
Here I find a solution:
Profiles file: (myprofile. CONF) / / Note: type your key with "spark" Otherwise, the prop will be ignored
spark.myapp.input /input/path spark.myapp.output /output/path
launch
$SPARK_HOME/bin/spark-submit --properties-file mypropsfile.conf
How to call code: (the code inside)
sc.getConf.get("spark.driver.host") // localhost sc.getConf.get("spark.myapp.input") // /input/path sc.getConf.get("spark.myapp.output") // /output/path
The content of this article comes from the network collection of netizens. It is used as a learning reference. The copyright belongs to the original author.
THE END
二维码