Java – how to sort Horizontally Partitioned Data

I have a telco billing software system There are daily user telephone records Logs are divided by date (month) level Each partition is stored in a separate database and can be distributed over multiple instances

In the UI, the user will specify a date range The returned data can be sorted on any field The date range can span multiple partitions The application must support paging through date range data

I can't load too many records into memory for sorting Placing sorting in a query can only provide sorting data in one result set

So I need to sort the data from multiple partitions How to return sorting records from multiple sorted result sets to the UI?

Editor: after more analysis of this problem, we have more input There are also pagination requirements Therefore, we need to find a way to sort multiple result sets in real time

Solution

By relying on the ability of resultset to load limited data in memory, we can use dynamic comparator to provide solutions in Java The solution is to get the first record from each resultset, sort it in Java, and return the first element from the sorted data

Detailed solution:

First, we build a program that can give us a two-dimensional comparator according to the criteria selected on the screen

Second, we write an aggregateresultset wrapper on Dao, which is wrapped in resultsets of different partitions Note: these individual resultsets have been sorted according to the same criteria The aggregateresultset is then assigned to the dynamic comparator

This aggregateresultset will have a data structure to initialize the first element of each result set It will return the next element of the call to next () This element will be the first one created from the dynamic comparator During the next () call, we delete the element from the temporary data structure and insert the next element from the same result set in the temporary data structure In this way, aggregateresultset will return data in the expected order by merging / storing / sorting very limited data in Java

We hope we won't receive comparison problems because most of our sorting is numeric / string data