What language is Hadoop

What is Hadoop? Hadoop is a software platform for developing and running large-scale data processing. It is an open-source software framework implemented in Java language by appch to realize distributed computing of massive data in a cluster composed of a large number of computers

The core design of Hadoop framework is HDFS and MapReduce HDFS provides the storage of massive data, and MapReduce provides the calculation of data

The process of data processing in Hadoop can be simply understood according to the following figure: the results are obtained after data processing through Hadoop cluster

HDFS: Hadoop distributed file system, Hadoop distributed file system

Large files are divided into data blocks of 64M by default, which are distributed and stored in cluster machines

As shown in the figure below, the file data1 is divided into three blocks, which are distributed in different machines in the form of redundant images

MapReduce: Hadoop creates a task for each input split and calls map calculation. In this task, each record in the split is processed in turn. The map will output the results in the form of key -- value. Hadoop is responsible for sorting the output of the map according to the key value as the input of reduce. The output of the reduce task is the output of the whole job and saved on HDFS

Hadoop cluster is mainly composed of namenode, datanode, secondary namenode, jobtracker and tasktracker

As shown in the figure below:

Namenode records how the file is split into blocks and how these blocks are stored in those datenode nodes

Namenode also saves the running state information of the file system

The datanode stores the split blocks

Secondary namenode helps namenode collect the status information of file system operation

Jobtracker is responsible for running jobs and scheduling multiple tasktrackers when a task is submitted to the Hadoop cluster

Tasktracker is responsible for a map or reduce task

summary

The above is what language Hadoop is introduced to you by Xiaobian. I hope it will help you. If you have any questions, please leave me a message, and Xiaobian will reply to you in time. Thank you very much for your support for the programming tips website!