Java IO stream splits a file into multiple sub files code example

File splitting and merging is a common requirement. For example, when uploading a large file, you can first split it into small pieces and transfer it to the server before merging. In many large distributed file systems (such as Google's GFS and Taobao's TFs), files are divided or merged by block.

Here are the basic ideas:

If there is a large file, specify the split size (for example, cut by 1m)

step 1：

First, calculate the number of small files n finally divided according to the original file size and split size

step 2:

Create these n small files on disk

step 3:

Open multiple threads (number of threads = number of split files). In each thread, use the seek function of RandomAccessFile to locate the read pointer to the first position of each segment in the original file, and then read the specified size backward (i.e. partition block size), and finally write the corresponding partition file. Because of multi-threaded parallel processing, the speed of writing each small file is relatively fast.

The following code is to split a file into multiple sub files, each with a size of 100k

summary

The above is all about the code example of Java IO stream splitting a file into multiple sub files. I hope it will be helpful to you. Interested friends can continue to refer to other related topics on this site. If there are deficiencies, please leave a message to point out. Thank you for your support!