String garbage collection in Java: or why consume so much memory

Solved

I try to understand why one of my unit tests consumes so much memory The first thing I do is run a test and measurement with visualvm:

The initial flat line is due to the thread at the beginning of the test Sleep() gives visualvm time to start

The test (and setup method) is very simple:

@BeforeClass
private void setup() throws Exception {
    mockedDatawireConfig = mock(DatawireConfig.class);
    when(mockedDatawireConfig.getUrl()).thenReturn(new URL("http://example.domain.fake/"));
    when(mockedDatawireConfig.getTid()).thenReturn("0000000");
    when(mockedDatawireConfig.getMid()).thenReturn("0000000");
    when(mockedDatawireConfig.getDid()).thenReturn("0000000");
    when(mockedDatawireConfig.getAppName()).thenReturn("XXXXXXXXXXXXXXX");
    when(mockedDatawireConfig.getNodeId()).thenReturn("t");

    mockedVersionConfig = mock(VersionConfig.class);
    when(mockedVersionConfig.getDatawireVersion()).thenReturn("000031");

    defaultCRM = new ClientRefManager();
    defaultCRM.setVersionConfig(mockedVersionConfig);
    defaultCRM.setDatawireConfig(mockedDatawireConfig);
}

@Test
public void transactionCountertest() throws Exception {
    Thread.sleep(15000L);
    String appInstanceID = "";
    for (Long i = 0L; i < 100000L; i++) {
        if (i % 1000 == 0) {
            Assert.assertNotEquals(defaultCRM.getAppInstanceID(),appInstanceID);
            appInstanceID = defaultCRM.getAppInstanceID();
        }
        ReqClientID r = defaultCRM.getReqClientID(); // This call is where memory use explodes.
        Assert.assertEquals(getNum(r.getClientRef()),new Long(i % 1000));
        Assert.assertEquals(r.getClientRef().length(),14);
    }
    Thread.sleep(10000L);
}

The test is very simple: iterate 100k times to ensure that defaultcrm Getreqclientid() generates a correct reqclientid object with a valid counter between 000-999, and the randomization prefix changes correctly when flipping

defaultCRM. Getreqclientid () is where memory problems occur Let's take a look:

public ReqClientID getReqClientID() {
    ReqClientID req = new ReqClientID();
    req.setDID(datawireConfig.getDid()); // #1
    req.setApp(String.format("%s&%s",datawireConfig.getAppName(),versionConfig.toString())); // #2
    req.setAuth(String.format("%s|%s",datawireConfig.getMid(),datawireConfig.getTid())); // #3

    Long c = counter.getAndIncrement();
    String appID = appInstanceID;
    if(c >= 999L) {
        LOGGER.warn("Counter exceeds 3-digits. Resetting appInstanceID and counter.");
        resetAppInstanceID();
        counter.set(0L);
    }
    req.setClientRef(String.format("%s%s%03dV%s",datawireConfig.getNodeId(),appID,c,versionConfig.getDatawireVersion())); // #4
    return req;
}

Very simple: create an object, call some string setters, calculate an up counter, and the random prefix on the rollover

Suppose I commented out setters (associated assertions, so they didn't fail), numbered #1-#4 Memory usage is now reasonable:

Initially, I used simple string concatenation in the setter component I changed it to string Format(), but it has no effect I also tried to use the string builder of append(), which is invalid

I also tried some GC settings In particular, I have tried - XX: useg1gc, - XX: initiating heapcoccupancypercent = 35, and - xms1g - xmx1g (note that 1g is still unreasonable on my buildlave, and I want it to drop at a maximum of about 256M) Here is the chart:

Down to - xms25m - xmx256m will cause outofmemoryerror

For the third reason, I am confused about this behavior First, I don't understand the extreme growth of unused heap space in the first figure I create an object, create some strings, pass the string to the object, and delete the object by making it out of range Obviously, I don't want to completely reuse memory, but why does the JVM seem to allocate more heap space for these objects every time? The way unused heap space grows so much faster seems really wrong Especially for more aggressive GC settings, I want to see the JVM try to reclaim these completely unreferenced objects before browsing memory

Second, in figure # 2, it is clear that the real problem is strings I've tried some reading about combining strings, text / internships, etc., but I can't see anything except / string Many options other than format () / StringBuilder seem to produce the same result Did I miss some magic ways to build strings?

Finally, I know that 100k iteration is excessive. I can test flip with 2K, but I try to understand what is happening in the JVM

System: openjdk x86_ 64 1.8. 0_ 92 and hotspot x86_ 64 1.8. 0_ seventy-four

Edit:

Several people have suggested that you call system. Net manually during testing GC (), so I try to execute every 1K cycle This has a significant impact on memory usage and has a serious impact on Performance:

The first thing to note is that although the heap space used grows slowly, it is still infinite The only time it is completely stable is the Thread. after the completion of the call. sleep(). Several questions:

1) Why is the unused heap space still so high? During the first iteration of the loop, call system gc()(i％1000 == 0). This actually leads to a reduction in unused heap space Why does the total heap space not decrease after the first call?

2) Very roughly, perform five allocations per loop iteration: Inst clientreqid and four strings Each iteration of the loop forgets all references to all five objects During the whole test process, the total object basically remains stationary (only change ~ ± 5 objects) I still don't understand why when the number of active objects remains the same, system GC () is not more effective in keeping heap space constants in use

Edit 2: solved

@Jonathan pointed me in the right direction by asking mockeddatawireconfig This is actually a spring @ configurationproperties class (that is, spring loads data from yaml into the instance and connects the instance to where it is needed) In unit testing, I didn't use anything related to spring (unit testing, not integration testing) In this case, it is just a POJO with getters and setters, but there is no logic in the class

In any case, unit tests use mock versions, as you can see in setup () above I decided to switch to a real instance of the object instead of a simulation This completely solves the problem! It seems that there may be some problems with mockito, or it may be because I seem to use 2.0 2-beta. I will investigate further and contact the mockito developer if it is indeed an unknown problem

Look at dat's sweet picture:

Solution

Well, it depends on how the JVM allocates heap space It just sees huge memory consumption (and fast!) Increase, so allocate enough heap space without encountering OutOfMemoryException

As you have seen, you can change this behavior by playing with parameters You can also see that once the usage remains the same, the heap will not grow again (it stops at ~ 3G instead of growing until ~ 8G)

To really understand what's happening, you shouldn't do some printf debugging (which means commenting something and seeing what happens), but use your ide or other tools to check what's using memory

Doing so will display (for example): the string of 120K instances consumes 2gib or 1.5gib garbage and 500mib as strings Then you know clearly whether it's just a lazy collection (because a collection has an overhead) or if you have some references that still fly around (I'll say no because growth stops)

As a dirty @ r_ 502_ 1911 @, you can also add system GC () calls to enforce garbage collection to see if it can improve heap utilization (at the cost of CPU time, of course)