Data heterogeneous solution for cache consistency and cross server query

When your project data volume increases, you will usually encounter two situations. The first is to use cache to combat the high concurrency of the upper layer as much as possible, and the second is to use sub databases

Split tables against high concurrency in the upper layer... It's easy to force, but it's not so optimistic. You may not have a good solution to the problems introduced. Let's share the details below.

For example, in our thousands of people and thousands of faces system, about 400 data models will be automatically established for a merchant's store based on the dimensions of commodities and orders. Then, after the buyer places an order on Taobao, Taobao will push the order

When the order is delivered, it will circle around 400 models, so as to push SMS and e-mail that are more appropriate and in line with the buyer's behavior habits. This is a real business scenario. In order to deal with high concurrency, these models are naturally slow

It exists in the cache, and the models are poured into redis from DB. If a new model comes in, how can I notify redis to update the cache??? The usual approach is to update the model when adding it

redis。。。 Right, as shown below:

To put it simply, I can write the code at hand. I want high cohesion, so you have to be discouraged.

In addition to getting rid of everything, you may also encounter success in updating database and failure in updating redis, but others don't care, and the error log is still in other people's log system, so it's difficult for you to even

The cache consistency between the DB and the cache cannot be guaranteed. Can I change my mind at this time? I directly write a program to subscribe to the database binlog and analyze the curd operation of the model data from the binlog

According to the actual situation of these curds, the redis cache data can be updated. The first can be decoupled from the web, and the second can achieve a high degree of cache consistency. Therefore, the new architecture is like this.

I believe you can understand the above figure. The key point is the binlog processing program, which analyzes curd from binlog to update redis. In fact, the binlog program is the canal...

A slave disguised as MySQL constantly steals binlog logs from MySQL through the dump command, which perfectly realizes this requirement.

It is also mentioned at the beginning of this article that when the amount of data is large, there will be sub databases and sub tables, and even databases will be distributed to multiple servers. Now the e-commerce projects are all business driven by technology...

No one knows what the next business will be, so it will inevitably lead you to do some cross server join queries,

However, if your business is really important, maybe DBA will do data heterogeneity for you. The so-called data heterogeneity is

Aggregate multiple tables to be joined into a DB according to a certain dimension. Let you check.....

If you use canal to subscribe to binlog, you can transform it into the following architecture.

Well, the application scenario of canal has also been introduced to you. The most important thing is to understand this idea. For things that others can't handle, your value comes out.

Open binlog and change the format of binlog to row, so that the binary content of curd can be obtained,

Use the command to verify, and the expiration time of starting binlog is 30 days. By default, binlog does not expire, which may cause your disk to be full until it hangs.

Address of GitHub: https://github.com/alibaba/canal/releases

canal

The mode of canal is as follows: there may be multiple instances in a canal, that is, one instance can monitor one MySQL instance, and multiple instances can correspond to multiple servers

MySQL instance of. That is, a canal can monitor the multi machine MySQL under the sub database and sub table.

It is a global canal server configuration. The details are as follows. The parameters involve all aspects.

As it is a global configuration, pay attention to the above three red areas:

canal. Port = 11111 current canal server port number

canal. Destinations = example currently, an instance named example is opened by default. If you want to open multiple instances, just separate them with "," commas...

canal. instance. filter. regex = .*\\..* All tables of all DB under the MySQL instance are within the monitoring range.

This is the configuration of a specific instance of instances. All configurations not involved will be from canal Properties.

Just pay attention to the places marked in red above. When stealing binlog, you need to know the MySQL address, user name and password.

You should remember to configure the / canal / bin directory into the path of / etc / profile to facilitate quick opening. You can see from the figure below that port 11111 has been opened on CentOS.

The canal driver needs to obtain the following information from Maven warehouse: http://www.mvnrepository.com/artifact/com.alibaba.otter/canal.client/1.0.24 However, there are still many dependencies.

The following code makes a basic judgment on the curd of table to see if it can be perceived intelligently, and then the redis can be updated according to the actual situation...

Update operation

Insert operation

Delete operation

From the results, there's nothing wrong. There's a picture and a truth. Well, that's all for this article. It must be helpful for you in development~~~