Java – BufferedReader in multi-core environment
I have eight documents Each of them is about 1.7 GB I'm reading these files into a byte array, and the operation is fast enough
Then read each file as follows:
BufferedReader br=new BufferedReader(new InputStreamReader(new ByteArrayInputStream(data)));
When processing sequentially with a single core, it takes 60 seconds to complete However, when allocating calculations on eight separate cores, each file takes far more than 60 seconds
Because the data is in memory and no IO operation is performed, I think the time required for each core to process a file should not exceed 60 seconds Therefore, a total of 8 files should be completed in 60 seconds, but this is not the case
Did I miss something about the BufferedReader behavior? Or any reader used in the above code
It is worth mentioning that I am using this code to upload files first:
byte[] content=org.apache.commons.io.FileUtils.readFileToByteArray(new File(filePath));
All codes are as follows:
For each file read the file into a byte[] add the byte[] to a list end For For each item in the list create a thread and pass a byte[] to it end For
Solution
How do you actually "distribute calculations"? Is synchronization involved? You just create 8 threads to read 8 files?
What platform are you running (Linux, windows, etc.)? I saw the seemingly strange behavior from the windows scheduler, and then it moved a single process from the core to the core to try to balance the load between the cores This eventually leads to performance degradation, not just allowing a single core to use more than the rest of the core