Write a reusable distributed scheduling task management webui component based on quartz
premise
Small entrepreneurial teams give priority to cost saving no matter what scheme they choose. For the distributed timing scheduling framework, mature candidate schemes include xxl-job, easy scheduler, light task scheduler and elastic job, which have been used in the production environment before. But to build a highly available distributed scheduling platform, These frameworks (whether decentralized or not) require additional server resources to deploy the central scheduling management service instance, and sometimes even rely on some middleware such as zookeeper. Recall that I spent some time looking at quartz's source code to analyze its thread model, and thought that it can adopt a less recommended X-lock scheme based on MySQL (select for update locking) realize that a single trigger in the service cluster has only one node (the node with successful locking) can execute. In this way, it can only rely on the existing MySQL instance resources to realize distributed scheduling task management. Generally speaking, business applications with relational data storage requirements will have their own MySQL instances. In this way, a distributed scheduling management module can be introduced at almost zero cost. The initial scheduling is finalized on an overtime Saturday afternoon After the first step, it took several hours to build the wheel. The effects are as follows:
conceptual design
Let's talk about all the dependencies:
The dependencies of the project are as follows:
<dependencies>
<dependency>
<groupId>org.quartz-scheduler</groupId>
<artifactId>quartz</artifactId>
<exclusions>
<exclusion>
<groupId>com.zaxxer</groupId>
<artifactId>HikariCP-java7</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter</artifactId>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-context-support</artifactId>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-jdbc</artifactId>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-freemarker</artifactId>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>com.zaxxer</groupId>
<artifactId>HikariCP</artifactId>
<scope>provided</scope>
</dependency>
</dependencies>
UIKit and jQuery can directly use the existing CDN:
<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/uikit@3.2.2/dist/css/uikit.min.css"/>
<script src="https://cdn.jsdelivr.net/npm/uikit@3.2.2/dist/js/uikit.min.js"></script>
<script src="https://cdn.jsdelivr.net/npm/uikit@3.2.2/dist/js/uikit-icons.min.js"></script>
<script src="https://cdn.bootcss.com/jquery/3.4.1/jquery.min.js"></script>
Table design
After introducing quartz's dependency, in its org quartz. impl. You can see a series of DDLS under the JDBC jobstore package. Generally, pay attention to tables in the scenario of using mysql_ MysqL. SQL and tables_ MysqL_ innodb. Two SQL files are enough. InnoDB must be selected as the engine of the development specification MySQL of the author's team, so the latter is selected.
The scheduled task information in the application should be managed separately to facilitate the provision of unified query and change API. It is worth noting that quartz built-in tables use a large number of foreign keys, so try to add, delete and modify the contents of its built-in tables through the API provided by quartz. Do not operate manually, otherwise various unexpected faults may be caused.
The two new tables introduced include the scheduling task table schedule_ Task and schedule task parameter table_ task_ parameter：
CREATE TABLE `schedule_task`
(
`id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT PRIMARY KEY COMMENT '主键',`creator` VARCHAR(16) NOT NULL DEFAULT 'admin' COMMENT '创建人',`editor` VARCHAR(16) NOT NULL DEFAULT 'admin' COMMENT '修改人',`create_time` DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT '创建时间',`edit_time` DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP COMMENT '修改时间',`version` BIGINT NOT NULL DEFAULT 1 COMMENT '版本号',`deleted` tinyint NOT NULL DEFAULT 0 COMMENT '软删除标识',`task_id` VARCHAR(64) NOT NULL COMMENT '任务标识',`task_class` VARCHAR(256) NOT NULL COMMENT '任务类',`task_type` VARCHAR(16) NOT NULL COMMENT '任务类型,CRON,SIMPLE',`task_group` VARCHAR(32) NOT NULL DEFAULT 'DEFAULT' COMMENT '任务分组',`task_expression` VARCHAR(256) NOT NULL COMMENT '任务表达式',`task_description` VARCHAR(256) COMMENT '任务描述',`task_status` tinyint NOT NULL DEFAULT 0 COMMENT '任务状态',UNIQUE uniq_task_class_task_group (`task_class`,`task_group`),UNIQUE uniq_task_id (`task_id`)
) COMMENT '调度任务';
CREATE TABLE `schedule_task_parameter`
(
`id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT PRIMARY KEY COMMENT '主键',`task_id` VARCHAR(64) NOT NULL COMMENT '任务标识',`parameter_value` VARCHAR(1024) NOT NULL COMMENT '参数值',UNIQUE uniq_task_id (`task_id`)
) COMMENT '调度任务参数';
Parameters are uniformly stored in JSON strings, so a scheduling task entity corresponds to 0 or 1 scheduling task parameter entities. The problem that multiple applications use the same data source is not considered here. In fact, this problem should be based on different org. Org quartz. jobStore. Tableprefix implements isolation, that is, if different applications share the same database, or the quartz of each application uses different table prefixes to distinguish, or separate all scheduling tasks into the same application.
Working mode of quartz
When quartz designs the scheduling model, it actually schedules the trigger trigger. Generally, when scheduling the corresponding task job, it needs to bind the trigger and the scheduled task instance. Then, when the trigger reaches the trigger time point, it will be fired, and then call back the execute () method of the job instance associated with the trigger. It can be simply understood that triggers and job instances are many to many relationships. In short, it is like this:
In order to realize this many to many relationship, quartz defines Jobkey and triggerkey respectively for job (actually JobDetail) and trigger as their unique identifiers.
TriggerKey -> [name,group]
JobKey -> [name,group]
In order to reduce the maintenance cost, the author forcibly restricts the many to many binding relationship to one-to-one, and assimilates the triggerkey and Jobkey as follows:
JobKey,TriggerKey -> [jobClassName,${spring.application.name} || applicationName]
In fact, most of the scheduling related work is entrusted to org quartz. The scheduler completes, as shown in the following example:
public interface Scheduler {
......省略无关的代码......
// 添加调度任务 - 包括任务内容和触发器
void scheduleJob(JobDetail jobDetail,Set<? extends Trigger> triggersForJob,boolean replace) throws SchedulerException;
// 移除触发器
boolean unscheduleJob(TriggerKey triggerKey) throws SchedulerException;
// 移除任务内容
boolean deleteJob(JobKey jobKey) throws SchedulerException;
......省略无关的代码......
}
All I have to do is pass the schedule_ The task table manages the scheduled tasks of the service through org quartz. The API provided by scheduler transfers the specific operation of the task to quartz, and adds some extended functions. This module has been encapsulated into a lightweight framework named quartz Web UI kit, hereinafter referred to as kit.
Kit core logic analysis
All core functions of the kit are encapsulated in the module quartz Web UI kit core. The main functions include:
The webui part is simply written through FreeMarker, jQuery and UIKit, mainly including three pages:
templates
- common/script.ftl 公共脚本
- task-add.ftl 添加新任务页面
- task-edit.ftl 编辑任务页面
- task-list.ftl 任务列表
The core method of scheduling task management is quartzwebuikitservice#refreshscheduletask():
@Autowired
private Scheduler scheduler;
public void refreshScheduleTask(ScheduleTask task,Trigger oldTrigger,TriggerKey triggerKey,Trigger newTrigger) throws Exception {
JobDataMap jobDataMap = prepareJobDataMap(task);
JobDetail jobDetail =
JobBuilder.newJob((Class<? extends Job>) Class.forName(task.getTaskClass()))
.withIdentity(task.getTaskClass(),task.getTaskGroup())
.usingJobData(jobDataMap)
.build();
// 总是覆盖
if (ScheduleTaskStatus.ONLINE == ScheduleTaskStatus.fromType(task.getTaskStatus())) {
scheduler.scheduleJob(jobDetail,Collections.singleton(newTrigger),Boolean.TRUE);
} else {
if (null != oldTrigger) {
scheduler.unscheduleJob(triggerKey);
}
}
}
private JobDataMap prepareJobDataMap(ScheduleTask task) {
JobDataMap jobDataMap = new JobDataMap();
jobDataMap.put("scheduleTask",JsonUtils.X.format(task));
ScheduleTaskParameter taskParameter = scheduleTaskParameterDao.selectByTaskId(task.getTaskId());
if (null != taskParameter) {
Map<String,Object> parameterMap = JsonUtils.X.parse(taskParameter.getParameterValue(),new TypeReference<Map<String,Object>>() {
});
jobDataMap.putAll(parameterMap);
}
return jobDataMap;
}
In fact, any task trigger or change directly overwrites the corresponding JobDetail and trigger, so as to ensure that the scheduling task content and trigger are brand-new, and the next round of scheduling will take effect.
The task class is abstracted as abstractscheduletask, which carries task execution and a large number of extended functions:
@DisallowConcurrentExecution
public abstract class AbstractScheduleTask implements Job {
protected Logger logger = LoggerFactory.getLogger(getClass());
@Autowired(required = false)
private List<ScheduleTaskExecutionPostProcessor> processors;
@Override
public void execute(JobExecutionContext context) throws JobExecutionException {
String scheduleTask = context.getMergedJobDataMap().getString("scheduleTask");
ScheduleTask task = JsonUtils.X.parse(scheduleTask,ScheduleTask.class);
ScheduleTaskInfo info = ScheduleTaskInfo.builder()
.taskId(task.getTaskId())
.taskClass(task.getTaskClass())
.taskDescription(task.getTaskDescription())
.taskExpression(task.getTaskExpression())
.taskGroup(task.getTaskGroup())
.taskType(task.getTaskType())
.build();
long start = System.currentTimeMillis();
info.setStart(start);
// 在MDC中添加traceId便于追踪调用链
MappedDiagnosticContextAssistant.X.processInMappedDiagnosticContext(() -> {
try {
if (enableLogging()) {
logger.info("任务[{}]-[{}]-[{}]开始执行......",task.getTaskId(),task.getTaskClass(),task.getTaskDescription());
}
// 执行前的处理器回调
processBeforeTaskExecution(info);
// 子类实现的任务执行逻辑
executeInternal(context);
// 执行成功的处理器回调
processAfterTaskExecution(info,ScheduleTaskExecutionStatus.SUCCESS);
} catch (Exception e) {
info.setThrowable(e);
if (enableLogging()) {
logger.info("任务[{}]-[{}]-[{}]执行异常",task.getTaskDescription(),e);
}
// 执行异常的处理器回调
processAfterTaskExecution(info,ScheduleTaskExecutionStatus.FAIL);
} finally {
long end = System.currentTimeMillis();
long cost = end - start;
info.setEnd(end);
info.setCost(cost);
if (enableLogging() && null == info.getThrowable()) {
logger.info("任务[{}]-[{}]-[{}]执行完毕,耗时:{} ms......",cost);
}
// 执行结束的处理器回调
processAfterTaskCompletion(info);
}
});
}
protected boolean enableLogging() {
return true;
}
/**
* 内部执行方法 - 子类实现
*
* @param context context
*/
protected abstract void executeInternal(JobExecutionContext context);
/**
* 拷贝任务信息
*/
private ScheduleTaskInfo copyScheduleTaskInfo(ScheduleTaskInfo info) {
return ScheduleTaskInfo.builder()
.cost(info.getCost())
.start(info.getStart())
.end(info.getEnd())
.throwable(info.getThrowable())
.taskId(info.getTaskId())
.taskClass(info.getTaskClass())
.taskDescription(info.getTaskDescription())
.taskExpression(info.getTaskExpression())
.taskGroup(info.getTaskGroup())
.taskType(info.getTaskType())
.build();
}
// 任务执行之前回调
void processBeforeTaskExecution(ScheduleTaskInfo info) {
if (null != processors) {
for (ScheduleTaskExecutionPostProcessor processor : processors) {
processor.beforeTaskExecution(copyScheduleTaskInfo(info));
}
}
}
// 任务执行完毕时回调
void processAfterTaskExecution(ScheduleTaskInfo info,ScheduleTaskExecutionStatus status) {
if (null != processors) {
for (ScheduleTaskExecutionPostProcessor processor : processors) {
processor.afterTaskExecution(copyScheduleTaskInfo(info),status);
}
}
}
// 任务完结时回调
void processAfterTaskCompletion(ScheduleTaskInfo info) {
if (null != processors) {
for (ScheduleTaskExecutionPostProcessor processor : processors) {
processor.afterTaskCompletion(copyScheduleTaskInfo(info));
}
}
}
}
The target scheduling task class to be executed only needs to inherit abstractscheduletask to obtain these functions. In addition, the post processor of scheduling task scheduletaskexecutionpostprocessor refers to the design of beanpostprocessor and transactionsynchronization in spring:
public interface ScheduleTaskExecutionPostProcessor {
default void beforeTaskExecution(ScheduleTaskInfo info) {
}
default void afterTaskExecution(ScheduleTaskInfo info,ScheduleTaskExecutionStatus status) {
}
default void afterTaskCompletion(ScheduleTaskInfo info) {
}
}
Through this post processor, various functions such as task alert and task execution log persistence can be completed. The author has realized the built-in early warning function through scheduletaskexecutionpostprocessor, and abstracted an early warning policy interface alarmstrategy:
public interface AlarmStrategy {
void process(ScheduleTaskInfo scheduleTaskInfo);
}
// 默认启用的实现是无预警策略
public class NoneAlarmStrategy implements AlarmStrategy {
@Override
public void process(ScheduleTaskInfo scheduleTaskInfo) {
}
}
By overriding the bean configuration of alarmstrategy, you can obtain a custom alert policy, such as:
@Slf4j
@Component
public class LoggingAlarmStrategy implements AlarmStrategy {
@Override
public void process(ScheduleTaskInfo scheduleTaskInfo) {
if (null != scheduleTaskInfo.getThrowable()) {
log.error("任务执行异常,任务内容:{}",JsonUtils.X.format(scheduleTaskInfo),scheduleTaskInfo.getThrowable());
}
}
}
Through the user-defined reality of this interface, the author prints all alerts to the nail group within the team, and prints the execution time, status, time-consuming and other information of the task. Once an exception occurs, it will timely @ everyone, so as to timely monitor the health of the task and subsequent tuning.
Using kit projects
The project structure of quartz Web UI kit is as follows:
quartz-web-ui-kit
- quartz-web-ui-kit-core 核心包
- h2-example H2数据库的演示例子
- MysqL-5.x-example MysqL5.x版本的演示例子
- MysqL-8.x-example MysqL8.x版本的演示例子
If you just want to experience the functions of the kit, download the project directly and start the club in the H2 example module throwable. h2. example. H2app, and then visit http://localhost:8081/quartz/kit/task/list Just.
Based on the application of MySQL instance, mysql5 with more users is selected here The example of X is briefly explained. Because the wheel has just been built and has not passed the test of time, it has not been handed over to Maven's warehouse for the time being. Manual compilation is required here:
git clone https://github.com/zjcscut/quartz-web-ui-kit
cd quartz-web-ui-kit
mvn clean compile install
Introducing dependencies (only quartz Web UI kit core needs to be introduced, and quartz Web UI kit core depends on spring boot starter web, spring boot starter web, spring boot starter JDBC, spring boot starter freemaker and hikaricp):
<dependency>
<groupId>club.throwable</groupId>
<artifactId>quartz-web-ui-kit-core</artifactId>
<version>1.0-SNAPSHOT</version>
</dependency>
<!-- 这个是必须，MysqL的驱动包 -->
<dependency>
<groupId>MysqL</groupId>
<artifactId>mysql-connector-java</artifactId>
<version>5.1.48</version>
</dependency>
Add a configuration to implement quartzwebuikitconfiguration:
@Configuration
public class QuartzWebUiKitConfiguration implements EnvironmentAware {
private Environment environment;
@Override
public void setEnvironment(Environment environment) {
this.environment = environment;
}
@Bean
public QuartzWebUiKitPropertiesProvider quartzWebUiKitPropertiesProvider() {
return () -> {
QuartzWebUiKitProperties properties = new QuartzWebUiKitProperties();
properties.setDriverClassName(environment.getProperty("spring.datasource.driver-class-name"));
properties.setUrl(environment.getProperty("spring.datasource.url"));
properties.setUsername(environment.getProperty("spring.datasource.username"));
properties.setPassword(environment.getProperty("spring.datasource.password"));
return properties;
};
}
}
Because the loading order of some components is considered in the design of quartz Web UI kit core, and the importbeandefinitionregistrar hook interface is used, attribute injection cannot be realized through @ value or @ Autowired, because the processing order of these two annotations is relatively late. If you use mappercannerconfigurer of mybatis, you will understand the problem here. A DDL script has been sorted out in the quartz Web UI kit core dependency:
scripts
- quartz-h2.sql
- quartz-web-ui-kit-h2-ddl.sql
- quartz-MysqL-innodb.sql
- quartz-web-ui-kit-MysqL-ddl.sql
You need to execute quartz MySQL InnoDB in the target database in advance SQL and quartz Web UI kit MySQL DDL sql。 A relatively standard configuration file application Properties are as follows:
spring.application.name=MysqL-5.x-example
server.port=8082
spring.datasource.driver-class-name=com.MysqL.jdbc.Driver
# 这个local是本地提前建好的数据库
spring.datasource.url=jdbc:MysqL://localhost:3306/local?characterEncoding=utf8&useUnicode=true&useSSL=false
spring.datasource.username=root
spring.datasource.password=root
# freemarker配置
spring.freemarker.template-loader-path=classpath:/templates/
spring.freemarker.cache=false
spring.freemarker.charset=UTF-8
spring.freemarker.check-template-location=true
spring.freemarker.content-type=text/html
spring.freemarker.expose-request-attributes=true
spring.freemarker.expose-session-attributes=true
spring.freemarker.request-context-attribute=request
spring.freemarker.suffix=.ftl
Then you need to add a scheduling task class, which only needs to inherit the club throwable. quartz. kit. support. AbstractScheduleTask：
@Slf4j
public class CronTask extends AbstractScheduleTask {
@Override
protected void executeInternal(JobExecutionContext context) {
logger.info("CronTask触发,TriggerKey:{}",context.getTrigger().getKey().toString());
}
}
Then start the startup class of springboot, and then access http://localhost:8082/quartz/kit/task/list ：
Add a scheduled task through the left button:
The current task expression supports two types:
Other optional parameters are:
As for the task expression parameters, neither strict verification nor string trim processing is considered. It is necessary to enter a compact specific expression in accordance with the agreed format, such as:
cron=*/20 * * * * ?
intervalInMilliseconds=10000
intervalInMilliseconds=10000,repeatCount=10
The scheduling task also supports the input of user-defined parameters. At present, the simple convention is JSON string. This string will be processed by Jackson and stored in the jobdatamap of the task. In fact, it will be persisted to the database by quartz:
{"key":"value"}
This can be obtained from jobexecutioncontext #getmergedjobdatamap(), for example:
@Slf4j
public class SimpleTask extends AbstractScheduleTask {
@Override
protected void executeInternal(JobExecutionContext context) {
JobDataMap jobDataMap = context.getMergedJobDataMap();
String value = jobDataMap.getString("key");
}
}
other
There are two points about kit design that the author has made special treatment based on the scenarios faced by the projects maintained in the team:
If you can't stand these two points, do not use this kit directly in production.
Summary
This paper briefly introduces that the author has built a lightweight distributed scheduling service wheel through the blessing of quartz, which is simple to use and cost-effective. The disadvantage is that considering that the services required for scheduling tasks in the current team's projects are internal shared services, the author does not spend much energy on improving authentication, monitoring and other modules. Here, it is also considered from the current business scenarios. If too many designs are introduced, it will evolve into a heavyweight scheduling framework, such as elastic job, That would go against the original intention of saving deployment costs.
(end of this article c-14-d e-a-20200410 is too busy recently. This article has been held back for a long time...)
The official account of Technology (Throwable Digest), which is not regularly pushed to the original technical article (never copied or copied):