Brotli compression multithreading

My understanding is that brotli stores the block size information in the meta block header, only the final uncompressed size of the block, and there is no information about the compressed length (9.2) I guess you need to create a wrapper to use it for multiple threads, or it might be similar to mark Adler's pigz

Do the same threading principles apply to brotli, just as gzip is used in this case, or do you have any foreseeable problems with multithreading implementation?

Solution

You can use the brotli format for this purpose I asked them to add the option to put metadata into an empty metablock (where "empty" means that the metablock produces zero uncompressed data) You can put tags in metadata to help find meta blocks The inserted empty block also starts the next block at the byte boundary

Each metablock can be independent of other metablocks If streams are constructed in this way, there is no problem compressing or decompressing them separately The areas that may depend on are the ring buffer of the last four distances used and the backward reference starting from the current metablock For parallel use, a metablock can and must be constructed so that it does not depend on the last four distances, rather than pointing to the ring buffer until it has filled the distance from the current metablock In addition, the distance returned before the current metablock (excluding static references) is not allowed Finally, you attach an empty metadata or metadata block to bring the sequence to the byte boundary for connection

By the way, it looks like you're linking to the old version in draft format This is a link to the current version