Read the storm operation zookeeper cluster clj

The main functions of storm operation zookeeper are defined in the namespace backtype storm. In cluster (i.e. in cluster.clj file). Backtype.storm.cluster defines two important protocols: clusterstate and stormclusterstate. @ h_502_4@

The protocol in clojure can be regarded as an interface in Java, encapsulating a set of methods. The clusterstate protocol encapsulates a set of basic functions that interact with zookeeper, such as obtaining child node functions, obtaining child node data functions, etc. the definition of the clusterstate protocol is as follows: @ h_ 502_ 4@

Clusterstate protocol @ h_ 502_ 4@

@H_ 502_ 4@

Stormclusterstate protocol encapsulates a set of functions for interacting between storm and zookeeper. Functions in stormclusterstate protocol can be regarded as "combinations" of functions in clusterstate protocol. The stormclusterstate protocol is defined as follows: @ h_ 502_ 4@

Stormclusterstate protocol @ h_ 502_ 4@

Namespace backtype storm. In addition to defining two important protocols, clusterstate and stormclusterstate, cluster also defines two important functions: MK distributed cluster state and MK storm cluster state@ H_ 502_ 4@

The MK distributed cluster state function is as follows: @ h_ 502_ 4@

This function returns an object that implements the clusterstate protocol. Through this object, you can interact with zookeeper@ H_ 502_ 4@

MK distributed cluster state function @ h_ 502_ 4@

The MK storm cluster state function is defined as follows: @ h_ 502_ 4@

The MK storm cluster state function is very important. It returns an instance that implements the stormclusterstate protocol. Through this instance, storm can interact with zookeeper more conveniently@ H_ 502_ 4@

The MK storm cluster state function is called in the functions that start nimbus and supervisor. The startup of nimbus and supervisor will be introduced in later articles@ H_ 502_ 4@

MK storm cluster state function @ h_ 502_ 4@

zookeeper. MK client function @ h in clj_ 502_ 4@

The MK client function creates a curatorframework instance and registers the curatorlistener for this instance. When a background operation is completed or the specified watch is triggered, the eventreceived() in the curatorlistener will be executed. The Wacher function invoked in eventReceived is the function in mk-distributed-cluster-state: the function of watcher binding. @ H_ 502_ 4@

The above is the source code analysis of the interaction between storm and zookeeper. I think the most important part is how to add "watcher" to ZK client. Many functions of storm are realized through zookeeper's watcher mechanism, such as "distribution information collection". Adding "Wacher" is roughly divided into the following steps: @ h_ 502_ 4@

The MK distributed cluster state function creates a ZK client and assigns a "watcher" function to the ZK client through the watcher. This "watcher" function simply calls the functions in the callbacks collection of clusterstate. In this way, the functions executed by the "watcher" function will be determined by the clusterstate instance. The clusterstate instance provides the register function to update the callbacks collection, The ClusterState instance is passed to the mk-storm-cluster-state function. Calling register in mk-storm-cluster-state adds a function (FN [type path]...). This function implements all the logic of the "watcher" function. The specific execution content of the functions registered in MK storm cluster state is determined by the stormclusterstate instance. Adding "observation" to the zookeeper node is also realized through the stormclusterstate instance, so that we can add "observation" and "callback function" to the nodes we are interested in through the stormclusterstate instance, When the node or node data changes, the ZK server will send a "notification" to the ZK client, and the "other" function in the ZK client will be called, and then the "return function" registered by us will be executed@ H_ 502_ 4@

Summary @ h_ 502_ 4@

This part of the source code is closely related to zookeeper and involves many concepts and features in zookeeper, such as "data observation" and "node observation". For zookeeper's Wacher mechanism, please refer to: https://www.oudahe.com/p/37336/ However, instead of directly using zookeeper's API, storm uses the cursor framework, which simplifies the operation of accessing zookeeper. For the cursor framework, refer to: https://www.oudahe.com/p/39023/ 。@ H_ 502_ 4@

The above is the storm operation zookeeper cluster Clj. For interested friends, please refer to: understanding of zookeeper watch mechanism, detailed explanation of Apache zookeeper usage examples, configuring corresponding ACL permissions for zookeeper, etc. I hope it will be helpful to you. Thank you for reading@ H_ 502_ 4@