premise

There was a plan to read AMQP in detail and pick out the key contents before learning rabbitmq. Later, I found the official document of rabbitmq and found a document that specifically introduced the AMQP model implemented in rabbitmq, so I wrote this article directly based on this document and personal understanding.

AMQP protocol

The full name of AMQP is advanced message queuing protocol. It is a (distributed) messaging protocol. Clients using and conforming to this protocol can communicate based on the messaging middleware agent (broker, i.e. broker, which is called proxy convergence) using and conforming to this protocol. AMQP has launched protocol 1.0. The well-known products to realize this protocol include stormmq, rabbitmq, Apache qpid, etc. The AMQP version implemented by rabbit MQ is 0.9 1. PDF Text download of the agreement is also provided in the official document. If you are interested, you can read it.

Responsibilities of message middleware agent

Messaging broker is called message middleware agent here. Its responsibility is to receive messages from publishers (publishers, or sometimes producers, producers) and then route the messages to consumers (consumers, or sometimes listeners).

Because the message middleware agent, publisher client and consumer client are based on the network message protocol AMQP, the message middleware agent, publisher client and consumer client can be on different machines to realize distributed communication and service decoupling.

Message middleware agent not only provides the two basic functions of message receiving and message routing, but also other advanced features, such as message persistence, monitoring and so on.

Basic model of amqp-0-9-1 in rabbitmq

The basic view of the amqp-0-9-1 model is: the message publisher publishes messages to the exchange. The role of the exchange is somewhat similar to the daily post office or mailbox. Then, the switch distributes the copy of the message to the queue. The rule followed when distributing the message is called binding. Then, the message middleware agent sends messages to consumers who subscribe to the queue (push mode), or consumers can actively pull messages from the queue (fetch / pull mode).

When publishing a message, the publisher can specify message attributes (message metadata). Some message metadata may be used by the message middleware agent, while other message metadata is opaque to the message middleware agent and is only used by message consumers.

Because the network is unreliable, the client may not be able to receive the message or fail to process the message. At this time, the message middleware agent cannot perceive whether the message is correctly delivered to the consumer. Therefore, the AMQP model provides the concept of message ack nowledge: when the message is delivered to the consumer, The consumer can automatically confirm that the message has been received successfully to the message middleware agent, or the application developer can choose to manually confirm that the message has been received successfully and confirm the message to the message middleware agent. The message middleware agent will completely delete the message from the queue only after receiving the confirmation notification of the message (or message group).

In some cases, if the switch cannot route to the queue correctly, the message will be returned to the publisher or discarded, or if the message middleware agent implements the "dead letter queue" extension, the message will be placed in the dead letter queue. The message publisher can choose to use the corresponding parameters to control the processing strategy of routing failure.

Exchanger and exchanger type

Exchange is the first destination of message sending. Its role is to receive messages and route them to zero or more queues. The algorithm for routing messages depends on the type of interactor and routing rules (that is, binding). Rabbitmq message middleware agent supports four types of interleavers:

When declaring the interactor, you need to provide some column attributes, among which the more important attributes are as follows:

Durability and auto delete features exist because all scenarios and use cases of concurrency require that the interactor is persistent.

Direct switch

Direct type switches deliver messages to the queue based on the message routing key. Direct switch is an ideal implementation of message unicast routing (of course, it can also be used for multicast routing). Its working principle is as follows:

Default switch

The default exchange is a special direct interactor. Its name is an empty string (that is, ""), which is pre declared by the message oriented middleware agent. In rabbitmq broker, its name in the web management interface is (AMQP default). Each newly created queue will be bound to the default switch. The routing key is the queue name of the queue, that is, all queues can deliver messages through the default switch. You only need to specify the routing key as the corresponding queue name.

Fanout exchanger

Fanout is actually a combined word. Fan means fan, and out means outward divergence. Fanout exchanger can be imagined as a "fan" exchanger. The fanout switch ignores the routing key and routes messages to all queues bound to it. That is, if n queues are bound to a fanout switch, when a new message is published to the fanout switch, a copy of the new message will be distributed to the N queues. Fanout switch is an ideal implementation of message broadcast routing.

Topic switch

The topic switch routes messages to one or more queues based on the routing key and the pattern of the binding queue and switch. The topic mode of the binding queue and switch (this mode string is actually the routing key when declaring the binding, which is not the same as the routing key of the message publishing) is generally separated by a dot (dot, that is'. '), such as source target. Key, the binding mode supports wildcards:

For each message, the topic switch will traverse all binding relationships, check whether the routing key specified by the message matches the routing key in the binding relationship, and if so, push the message to the corresponding queue.

Topic switch is an ideal implementation of message multicast routing.

Headers exchanger

Headers switch is an uncommon switch. It uses multiple attributes for routing. These attributes are generally called message headers. It does not use routing keys for message routing. Message headers are part of message properties (message metadata). Therefore, when using headers switch to establish the binding relationship between queue and switch, you need to specify a group of key value pairs. When sending messages to headers switch, you need to carry a group of key value pairs in message properties as message headers. The message header attribute supports the matching rule x-match as follows:

The headers switch also ignores the routing key and only relies on the message header in the message attribute for message routing.

Queues in the AMQP 0-9-1 model are very similar to queues in other message or task queuing systems: they store messages used by applications. The basic properties of queues and switches are similar:

A queue can only be used when it is declared, that is, the first declaration of the queue is the creation of the queue (because the queue does not exist when it is declared for the first time). If the existing queue is declared again with the same parameters, the declaration will not take effect (of course, there will be no exceptions). However, if an existing queue is declared again with different parameters, a channel level exception will be thrown. The exception code is 406 (precondition_failed).

Queue name

The queue name must consist of UTF-8 encoded characters within 255 bytes. The message middleware agent implementing AMQP 0-9-1 specification has the function of automatically generating random queue names, that is, when declaring a queue, if the queue name is specified as an empty string, the message middleware agent will automatically generate a queue name and bring the corresponding queue name in the return result of the queue declaration.

With "AMQ." The queue at the beginning is generated internally by the message middleware agent and has a special function. Therefore, a new queue with this name cannot be declared, otherwise it will lead to channel level exception, and the exception code is 403 (access_refreshed).

Persistence characteristics of queues

The persistent queue will be persisted to the disk. This queue will not be deleted after the message middleware agent is restarted. A queue that does not enable the persistence feature is called a transient queue. Not all scenarios need to enable the persistence feature of the queue.

The persistence feature of a queue does not mean that the messages routed to it are persistent, that is, the persistence of a queue is different from the persistence of messages. If the Information Middleware agent hangs, it will re declare the queues with the persistence feature enabled after restart. Only the messages in these queues that use the message persistence feature will be recovered.

binding

Binding is the rule that the switch routes messages to the queue. For example, if switch e can route messages to queue Q, Q must be bound to e through certain rules. The type of some switches used in the binding determines that it can use an optional routing key. The function of routing key is similar to that of filter, which can filter some messages published to the switch and route them to the target queue.

If the published message is not routed to any target queue, for example, the message has been published to the switch and there is no binding in the switch, the message will be discarded or returned to the publisher, depending on the parameters used by the publisher when publishing the message.

consumer

If only publishers produce messages in the queue, it is meaningless. There must be consumers to use messages, or this operation is called message consumption. There are two ways of message consumption:

In the case of push mode, the consumer must specify the queue to subscribe to. There can be multiple consumers per queue, or only one exclusive consumer can be registered.

Each consumer (subscriber) has an identifier called consumer tag, which is a string. You can unsubscribe through the consumer tag.

Message confirmation

Consumer applications may crash when receiving and processing messages, or the message middleware agent may fail to deliver messages to consumers due to network reasons. This will give rise to a problem: when should AMQP message middleware agent delete messages from the queue? Therefore, AMQP 0-9-1 specification provides two options:

The former is called automatic confirmation model (message confirmation is performed when the action is triggered), and the latter is called explicit confirmation model. In the explicit confirmation model, the consumer needs to actively confirm the message to the message middleware agent. The execution time of this message active confirmation action is completely controlled by the application. There are three ways of message active acknowledgement: positive acknowledgement (ACK), negative acknowledgement (NACK) and reject.

Prefetch message

Prefetching messages is a feature. When multiple consumers share the same queue, the message middleware agent can be informed to specify the amount of messages that each consumer can receive at one time before sending the next confirmation. This feature can be understood as a simple load balancing technology, which can improve throughput in the scenario of batch message publishing.

Message properties and payload

In the AMQP model, messages have attribute values. AMQP 0-9-1 specification defines some common attributes, which general developers do not need to pay much attention to:

These common attributes are generally used by the message oriented middleware agent, and there is also an optional attribute header that can be customized in the form of key value pairs, similar to the request header in http. Message properties are set when publishing messages.

The AMQP message also has a payload (which is actually the message data body). The AMQP agent treats it as an opaque byte array, that is, the AMQP agent will not check or modify the payload of the message. Some messages may contain only properties and no payload. Serialization formats (such as JSON, thrift, protocol buffers, and messagepack) are often used to serialize and structure data so that it can be published as a message payload. Under the general convention, the content type and content encoding in the message attribute can generally indicate the serialization method.

Message publishing supports the message persistence feature. After the message persistence feature is enabled, the message middleware agent will save the message to disk. If the agent is restarted, the message will not be lost. Turning on the message persistence feature will affect performance, mainly because it involves disk brushing.

Amqp-0-9-1 method

AMQP 0-9-1 defines some methods, corresponding to some operation methods of interaction between client and message middleware agent. The design of these operation methods has nothing in common with the methods in object-oriented programming language. Common operation methods related to exchanger include:

Logically, the above operation methods interact between the client and the message middleware agent as follows:

There are similar operation methods for queues:

Not all AMQP operation methods have response result operation methods, such as message publishing method basic Publish is the most widely used. This operation method has no corresponding response result operation method. Some operation methods may have multiple response results (operation methods), such as basic get。

Connection

AMQP connections usually exist for a long time. AMQP is an application level protocol that uses TCP for reliable delivery. AMQP connections use user authentication and can be protected using TLS (SSL). When an application no longer needs to connect to the AMQP proxy, it should normally close the AMQP connection instead of suddenly closing the underlying TCP connection.

Channel

Some applications require multiple connections to the AMQP agent. However, you do not want to open many TCP connections at the same time, because doing so will consume system resources and make it very difficult to configure the firewall. A channel can be considered as a "lightweight connection sharing a single TCP connection". An AMQP connection can have multiple channels.

For applications that use multithreading, there is a common usage scenario: each thread opens a new channel, which is isolated between threads.

In addition, each specific channel and other channels are isolated from each other. Each executed AMQP operation method (including response) carries the unique identification of a channel, so that the client can know which channel the operation method corresponds to through the unique identification of the channel.

Virtual host

In order to enable a single message middleware agent to host multiple completely isolated "environments" (isolation here refers to user groups, interactors, queues, etc.), AMQP provides the concept of virtual host. Multiple virtual hosts are similar to the virtual hosts of many mainstream web servers, providing a completely isolated environment for AMQP components. The AMQP client can specify the virtual host to be connected when connecting to the message middleware agent.

Personal understanding

About exchange, queue, and binding

To understand the AMQP model in rabbitmq, in fact, from the perspective of developers, the most important thing is the relationship between exchange, queue and binding. Here are my personal opinions. The first stop of message publishing is always exchange. From the model, message publishing cannot be directly sent to the queue. Exchange itself does not store messages. After receiving messages, it will route messages to the target queue based on routing rules, that is, binding. In practice, the routing rules are always declared before publishing and consuming messages, that is, the general steps are as follows:

In the two phases we are most concerned about, message publishing and message consumption, message publishing is actually only related to exchange, while message consumption is actually only related to queue. Binding is actually the contractual relationship between exchange and queue, which will directly affect the message routing in the message publishing phase. So, what is the general cause of routing failure? Routing failure is actually that the message has been published to exchange, and the existing target queue cannot be found from the existing binding in exchange to deliver the message copy (generally speaking, few people send messages to a nonexistent exchange). The failure of message routing can be fundamentally avoided by understanding the AMQP model, unless it is caused by the deliberate indiscriminate use or human error of the message publisher using the nonexistent routingkey, exchange or binding relationship.

About exchange types

Amqp-0-9-1 model supports four switches: direct, fanout, topic and headers. In fact, from the user's point of view, the functions of the four switches can be replaced by each other. For example, you can use a fan out switch to broadcast. In fact, you can also use a direct switch to broadcast, but the corresponding direct switch needs to be bound to multiple target queues through multiple routing keys. When selecting the technology for the production environment, we need to consider the performance, maintenance difficulty, rationality and other aspects to consider what type of switch to choose. For the above example of broadcast messages, it is obvious that using fanout type switches can avoid declaring multiple binding relationships, which is a better choice in terms of performance and rationality.

About load balancing

In the amqp-0-9-1 model, the implementation of load balancing is based on consumers rather than queues (specifically, it should be the way messages are delivered to queues). In fact, when the message production speed greatly exceeds the consumption speed of consumers, there may be a message backlog in the queue. The amqp-0-9-1 model does not provide the feature of queue based load balancing, that is, when the message production speed greatly exceeds the consumption speed of consumers, the message will not be routed to multiple queues, but the consumption capacity of the message maker is determined through the feature of prefetching messages, So as to adjust the number of messages pushed by the message middleware agent to the corresponding consumers, so that consumers with fast consumption speed can consume more messages, and reduce the problems that some consumers are hungry and some consumers are busy for a long time.

About message confirmation mechanism

The message confirmation mechanisms provided in AMQP mainly include positive confirmation (generally called ACK, ACK nowledgement), negative confirmation (generally called NACK, negative ACK nowledgement) and reject. Message confirmation mechanism is an important measure to ensure that messages are not lost. When consumers receive a message pushed by the message middleware agent, they need to actively notify the message middleware agent that the message has been confirmed to be delivered successfully, and then the message middleware agent will delete the corresponding message from the queue. Messages without active confirmation will change to "NACK" status. It can be imagined that they are temporarily stored in the "NACK area" of the queue. These messages will not be delivered to the consumer. Until the consumer restarts, the messages in the "NACK area" will change to "ready" status again and can be delivered to the consumer again. In fact, the scenario of message confirmation mechanism is complex, which will be analyzed in a later article.

Summary

reference material:

Personal blog

(end of this paper e-a-20181125 c-7-d)

The official account of Technology (Throwable Digest), which is not regularly pushed to the original technical article (never copied or copied):

Entertainment official account ("sand sculpture"), select interesting sand sculptures, videos and videos, push them to relieve life and work stress.