Detailed explanation of standalone mode deployment configuration of spark
Spark operation mode
Spark has many modes, the simplest of which is the single machine local mode and the single machine pseudo distributed mode. The complex ones run in the cluster. At present, they can run well in yarn and mesos. Of course, spark also has its own standalone mode. For most cases, the standalone mode is enough. If the enterprise already has a yarn or mesos environment, It is also very easy to deploy.
1. Local (local mode): it is commonly used for local development and testing. Local is also divided into local single thread and local cluster multi thread;
2. Standalone mode: a typical mater / slave mode, but it can also be seen that the master has a single point of failure; Spark supports zookeeper to implement ha
3. On yarn (cluster mode): it runs on the yarn resource manager framework. Yarn is responsible for resource management, and spark is responsible for task scheduling and calculation
4. On mesos (cluster mode): running on the mesos resource manager framework, mesos is responsible for resource management, and spark is responsible for task scheduling and calculation
5. On cloud (cluster mode): for example, EC2 of AWS can easily access Amazon S3 by using this mode; Spark supports a variety of distributed storage systems: HDFS and S3
Spark standalone cluster deployment
preparation
1. Here I download the compiled version of spark, otherwise I need to compile it myself in advance
2. Spark requires HDFS of Hadoop as the persistence layer, so Hadoop needs to be installed before spark is installed. The installation of Hadoop is not introduced here. A tutorial on Hadoop stand-alone installation and configuration is given
The above is all about the detailed explanation of the deployment and configuration of Spark's standalone mode in this article. Interested friends can continue to refer to: talking about seven common Hadoop and spark project cases, explaining in detail the methods of writing and running spark applications in Java, and the code examples of Spark's broadcast variables and adder usage methods, hoping to help you. Thank you for your support for the programming tips website!