In depth understanding of idempotency and detailed explanation of idempotency of restful style API

What is idempotency

The definition of idempotency in http / 1.1 is that a single and multiple requests for a resource should have the same result for the resource itself (except for network timeout). That is, the impact of any multiple execution on the resource itself is the same as that of a single execution. Here we need to pay attention to several key points: idempotency is not just one time (or multiple) requests have no side effects on resources (for example, the query database operation does not add, delete or change, so it has no impact on the database). Idempotence also includes the side effects on resources when the first request is made, but subsequent requests will not have side effects on resources. Idempotence focuses on whether subsequent requests have side effects on resources, rather than the results. Network timeout, etc Problem is not the scope of idempotent discussion. Idempotency is a commitment (rather than implementation) of system services to the outside world. It promises that as long as the interface is successfully called, the impact of multiple external calls on the system will be the same. Services declared as idempotent will think that external call failure is normal, and there will be retries after failure.

When is idempotency required

In business development, repeated submissions are often encountered, whether the request is re initiated due to network problems and unable to receive the request results, or the repeated submission is caused by the operation jitter of the front end. In the transaction system, the problems caused by repeated submission in the payment system are particularly obvious, such as:

If the user clicks on the app repeatedly to submit an order, only one order should be generated in the background;

Alipay launched a payment request, Internet problem or system BUG reissued, Alipay should only deduct one time. Obviously, the service declaring idempotent believes that there will be multiple calls by external callers. In order to prevent multiple changes to the system data state caused by multiple external calls, the service is designed as idempotent.

Discrimination between idempotence and weight prevention

In the above example, the problem encountered by Xiao Ming is only repeated submission, which is different from the original intention of service idempotence. Repeated submission refers to the artificial multiple operations when the first request has been successful, resulting in multiple changes in the state of the service that does not meet the idempotent requirements. Idempotent is more often used to initiate multiple requests when the first request does not know the result (such as timeout) or fails. The purpose is to confirm the success of the first request many times, but there will be no multiple state changes due to multiple requests.

Under what circumstances does idempotency need to be guaranteed

Taking SQL as an example, there are three scenarios. Only the third scenario requires developers to use other strategies to ensure idempotency:

Select col1 from tab1 where col2 = 2. It will not change the state no matter how many times it is executed. It is a natural idempotent.

Update tab1 set col1 = 1 where col2 = 2. The status is consistent no matter how many times the execution is successful, so it is also an idempotent operation.

Update tab1 set col1 = col1 + 1 where col2 = 2. The result of each execution will change. This is not idempotent.

Why design idempotent services

Idempotent can make the client logic processing simple, but at the cost of the complexity of the service logic. To meet the needs of idempotent services, there are at least two points in the logic:

First, query the last execution status. If not, it is considered as the first request

Before the service changes the business logic of the state, ensure the logic of anti duplicate submission

Idempotent deficiency

Idempotent is to simplify the logic processing of the client, but increases the logic and cost of the service provider. Whether it is necessary needs to be analyzed according to specific scenarios. Therefore, except for special business requirements, idempotent interfaces are not provided as far as possible.

The business logic of additional control idempotent is added to complicate the business function;

The function of parallel execution is changed to serial execution, which reduces the execution efficiency.

Strategies to guarantee idempotence

Idempotency needs to be guaranteed by a unique business order number. That is, the same business order number is considered to be the same business. This unique business order number is used to ensure that the processing logic and execution effect of the same business order number for many times are consistent. Taking payment as an example, it is easy to implement idempotence without considering Concurrency: ① first query whether the order has been paid; ② if it has been paid, it returns payment success; If there is no payment, proceed to the payment process and modify the order status to paid.

Policies to prevent duplicate submissions

The above guaranteed idempotent scheme is divided into two steps. Step ② depends on the query results of step ① and cannot guarantee atomicity. In the case of high parallel delivery, the following situation will occur: the second request comes when the order status in step ② of the first request has not been changed to "paid status". Now that this conclusion is reached, the remaining problems become simple: lock the query and change state operations, and change the parallel operation to serial operation.

Optimistic lock

If you only update the existing data, there is no need to lock the business. Optimistic locks are used when designing the table structure. Generally, optimistic locks are used through version, which can ensure both execution efficiency and idempotence. For example: update tab1 set col1 = 1, version = version + 1 where version = #version # however, the optimistic lock is invalid, which is often referred to as the ABA problem. However, if the version is always self increasing, ABA will not occur. If the version of version is not self incremented, we can add a timestamp to ensure that the timestamp field will not be updated, and verify the consistency of the timestamp. You can think so. Suppose we have two threads. The first thread gets version and the second thread gets version. Then, the first thread updates and increases the version at the same time, so that even if the second thread performs the update, it will not operate because the version has changed.

Anti weight meter

The order number orderNo is used as the unique index of the de duplication table. Each request inserts a piece of data into the de duplication table according to the order number. For the first time, the order payment status is requested to be queried. Of course, the order has not been paid. The payment operation is performed. Whether it is successful or not, the order status is updated to success or failure after execution, and the data in the de duplication table is deleted. If the subsequent order fails to insert because of the unique index in the table, the operation fails until the first request is completed (success or failure). It can be seen that the function of anti duplication table is to lock. It is carried out through the unique index. When the same data is inserted into the unique index table, an error will be reported. Example: create a test table

CREATE TABLE `test03` (
`id` INT(11),`uid` INT(11) DEFAULT NULL);

Create unique index

ALTER IGNORE TABLE test03 ADD UNIQUE INDEX  id_uid(id,uid);

Insert the data again and find that the same data cannot be inserted

INSERT INTO test03(id,uid) VALUES (1,1),(1,2),1);

In fact, this method is similar to the redis method below, and the method below is faster.

Distributed lock

The anti duplication table used here can be replaced by distributed locks, such as redis. When an order initiates a payment request, the payment system will query whether the key of the order number exists in redis cache. If it does not exist, the key will be added to redis as the order number. Query whether the order payment has been paid. If not, payment will be made. After payment, delete the key of the order number. Distributed locking is achieved through redis. Only after this order payment request is completed can the next request come in. Compared with the de duplication table, the concurrency is put into the cache, which is more efficient. The idea is the same, and only one payment request can be completed at a time.

Token token

This method is divided into two stages: the application token stage and the payment stage. In the first stage, before entering the order submission page, the order system needs to send a request for token to the payment system according to the user information, and the payment system saves the token in redis cache for use in the second stage of payment. In the second stage, the order system initiates a payment request with the applied token. The payment system will check whether the token exists in redis. If it exists, it means that the payment request is initiated for the first time. After deleting the token in the cache, the payment logic processing begins; If it does not exist in the cache, it indicates an illegal request. In fact, the token here is a token. The payment system confirms that you are your mother's child according to the token. The disadvantage is that it requires two interactions between systems, and the process is more complex than the above method.

Payment buffer

The payment request of the order is quickly followed by a buffer pipeline for quick receipt of the order. Subsequently, asynchronous tasks are used to process the data in the pipeline and filter out duplicate pending orders. The advantages are synchronous to asynchronous and high throughput. The disadvantage is that the payment results cannot be returned in time. It is necessary to monitor the asynchronous return of payment results.

Detailed explanation of restfulapi idempotency

We should note that idempotents are resource oriented. This HTTP get method may get different return contents each time, but it does not affect resources.

Get method

The HTTP get method is used to obtain resources. No matter how many times the interface is called, the result will not change, so it is idempotent. Just querying the data will not affect the change of resources, so we think it is idempotent.

Post method

The HTTP post method is a non idempotent method, because multiple calls will generate new resources. Because it will affect the resource itself, a new resource will be generated every call, so it does not meet idempotency.

Put method

Because it directly replaces the data of the entity part with the resources of the server, we call it many times, which will only have an impact once, but the HTTP method with the same result meets idempotence.

Delete method

The HTTP delete method is used to delete resources, which will be deleted. One and more calls have the same impact on resources, so they also meet idempotency. So for restful, only post is not enough.