Java – restriction of the original buffer – loading partial data and shared strings

I want to check if the raw buffer is the best serializer I use, and my research has found no other approach

message All {
    repeated Line lines = 1;
    Common common = 2;
}

There are hundreds of line objects, each line is quite complex, and you need ~ 100 kb

The two problems I see using the original buffer - I only need a small amount of available data when the application starts - just the basic information of "common" and "line" Can I load some data? – Each line object contains hundreds of strings, but the same string will appear in multiple line objects, so I want to try to share them among these objects Is it possible at the original buf level, or does it need to be part of the application level?

thank you!

Solution

It is difficult to give appropriate feedback from the limited specifications you give You said that the best solution to your problem seems to be protobuf, but we can't re evaluate it from the information given

According to what you wrote, I would even say that a simple gzipped byte array (or mostly printable JSON) (they can be said to be strings)) might be better ("reuse" many things of cross line objects = & gt; gzip will rock)

As others said: with protobuf, it is impossible to load "partial data structure" (in fact, you will not be a partial data structure), as long as the "repeated" part is a collection, because protobuf will allocate the data of your structure itself)

I will put my two cents gzipped JSON stream (for example, with Jackson). Of course, in this case, you will reduce the cost of bandwidth and CPU cycle