Java – how to handle thousands of quartz queries gracefully?

We have a required application

>Reprocess large amounts of data at night, and > reprocess large amounts of data as needed

In both cases, about 10000 quartz operations are generated and run At night, we have a quartz cron job, which can generate 10000 jobs, and each job processes data separately

Our problem is that we are running about 30 threads, so the quartz work will certainly catch fire and continue to catch fire until everything is processed The processing time is up to 6 hours Each of these 10000 jobs involves a specific domain object that can be processed in parallel and completely independent Each of the 10000 jobs can take a variable time (from half a second to a minute)

My question is:

Is there a better way to do this? > If not, our best way is to arrange / set up our quartz work so as to spend the least time, money and deal with the fire

A note about or architecture: we run two three node clusters The quartz version is a little old (2.0.1) and is in quartz Clustering is enabled in the properties file

Solution

There is no need to generate new quartz operations Quartz is a scheduler, not a task manager

During night reprocessing – you only need one quartz cron job to call some services responsible for managing and running 10000 tasks Quartz should not be involved in "on - demand" situations Call the service directly

How does the service manage 10000 tasks?

Typically, when only one JVM is available, you only need to use some executorservices Here, because there are 6 nodes under your finger, you can easily use hazelcast Hazelcast is a Java library that can be used for cluster nodes and share resources. Hazelcast has a simple solution for distributing your executorservice, that is, distributed executor service This is as simple as creating hazelcast executorservice and submitting the task on all members The following is an example of the document being called in a member:

Callable<String> task = new Echo(input); // Echo is just some Callable
HazelcastInstance hz = Hazelcast.newHazelcastInstance();
IExecutorService executorService = hz.getExecutorService("default");
Future<String> future = executorService.submitToMember(task,member);
String echoResult = future.get();