What language is Hadoop
What is Hadoop? Hadoop is a software platform for developing and running large-scale data processing. It is an open-source software framework implemented in Java language by appch to realize distributed computing of massive data in a cluster composed of a large number of computers
The core design of Hadoop framework is HDFS and MapReduce HDFS provides the storage of massive data, and MapReduce provides the calculation of data
The process of data processing in Hadoop can be simply understood according to the following figure: the results are obtained after data processing through Hadoop cluster
HDFS: Hadoop distributed file system, Hadoop distributed file system
Large files are divided into data blocks of 64M by default, which are distributed and stored in cluster machines
As shown in the figure below, the file data1 is divided into three blocks, which are distributed in different machines in the form of redundant images
MapReduce: Hadoop creates a task for each input split and calls map calculation. In this task, each record in the split is processed in turn. The map will output the results in the form of key -- value. Hadoop is responsible for sorting the output of the map according to the key value as the input of reduce. The output of the reduce task is the output of the whole job and saved on HDFS
Hadoop cluster is mainly composed of namenode, datanode, secondary namenode, jobtracker and tasktracker
As shown in the figure below:
Namenode records how the file is split into blocks and how these blocks are stored in those datenode nodes
Namenode also saves the running state information of the file system
The datanode stores the split blocks
Secondary namenode helps namenode collect the status information of file system operation
Jobtracker is responsible for running jobs and scheduling multiple tasktrackers when a task is submitted to the Hadoop cluster
Tasktracker is responsible for a map or reduce task
summary
The above is what language Hadoop is introduced to you by Xiaobian. I hope it will help you. If you have any questions, please leave me a message, and Xiaobian will reply to you in time. Thank you very much for your support for the programming tips website!