Why can’t this java process terminate?
I have an intermittent problem on the build server, where the java process under construction cannot be terminated and seems to continue running (using 100% CPU) forever (I have seen it run for 2 days on weekends, which takes about 10 minutes) Kill -9 PID seems to be the only way to stop a process
I've tried to call kill - quit PID on the process, but it doesn't seem to generate any stack trace to stdout (maybe it doesn't respond?) Jstack without the - f mandatory option does not seem to be able to connect to the running JVM, but using the force option will produce the output contained below
Unfortunately, even with stack traces, I don't see any obvious path to further investigation
As far as I know, it shows two "blocked" threads that run object Wait (their stack seems to contain only core Java code, we don't), the third is "in_vm", and there is no stack output
What steps should I take to gather more information about the cause of the problem (or better, how do I resolve it)?
$/opt/jdk1.6.0_29/bin/jstack -l -F 5546
Attaching to process ID 5546,please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 20.4-b02
Deadlock Detection:
No deadlocks found.
Finding object size using Printezis bits and skipping over...
Thread 5555: (state = BLOCKED)
Locked ownable synchronizers:
- None
Thread 5554: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.ref.ReferenceQueue.remove(long) @bci=44,line=118 (Interpreted frame)
- java.lang.ref.ReferenceQueue.remove() @bci=2,line=134 (Interpreted frame)
- java.lang.ref.Finalizer$FinalizerThread.run() @bci=3,line=159 (Interpreted frame)
Locked ownable synchronizers:
- None
Thread 5553: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.Object.wait() @bci=2,line=485 (Interpreted frame)
- java.lang.ref.Reference$ReferenceHandler.run() @bci=46,line=116 (Interpreted frame)
Locked ownable synchronizers:
- None
Thread 5548: (state = IN_VM)
Locked ownable synchronizers:
- None
(Java version 1.6.0 update 29, running on scientific Linux version 6.0)
to update:
Run strace - F - P 894 to generate a seemingly endless stream
[pid 900] sched_yield() = 0 [pid 900] sched_yield() = 0 ...
Then when Ctrl CD
Process 894 detached ... Process 900 detached ... Process 909 detached
Jmap - histo 894 does not connect, but jmap - F - histo 894 returns
Attaching to process ID 894,please wait... Debugger attached successfully. Server compiler detected. JVM version is 20.4-b02 Iterating over heap. This may take a while... Finding object size using Printezis bits and skipping over... Finding object size using Printezis bits and skipping over... Object Histogram: num #instances #bytes Class description -------------------------------------------------------------------------- 1: 11356 1551744 * MethodKlass 2: 11356 1435944 * ConstMethodKlass 3: 914 973488 * ConstantPoolKlass 4: 6717 849032 char[] 5: 16987 820072 * SymbolKlass 6: 2305 686048 byte[] 7: 914 672792 * InstanceKlassKlass 8: 857 650312 * ConstantPoolCacheKlass 9: 5243 167776 java.lang.String 10: 1046 108784 java.lang.Class 11: 1400 87576 short[] 12: 1556 84040 * System ObjArray 13: 1037 64584 int[] 14: 103 60152 * ObjArrayKlassKlass 15: 622 54736 java.lang.reflect.Method 16: 1102 49760 java.lang.Object[] 17: 937 37480 java.util.TreeMap$Entry 18: 332 27960 java.util.HashMap$Entry[] 19: 579 27792 java.nio.HeapByteBuffer 20: 578 27744 java.nio.HeapCharBuffer 21: 1021 24504 java.lang.StringBuilder 22: 1158 24176 java.lang.Class[] 23: 721 23072 java.util.HashMap$Entry 24: 434 20832 java.util.TreeMap 25: 689 18936 java.lang.String[] 26: 238 17440 java.lang.reflect.Method[] 27: 29 16800 * MethodDataKlass 28: 204 14688 java.lang.reflect.Field 29: 330 13200 java.util.LinkedHashMap$Entry 30: 264 12672 java.util.HashMap ... 585: 1 16 java.util.LinkedHashSet 586: 1 16 sun.rmi.runtime.NewThreadAction$2 587: 1 16 java.util.Hashtable$EmptyIterator 588: 1 16 java.util.Collections$EmptySet Total : 79700 8894800 Heap traversal took 1.288 seconds.
Solution
This may be due to insufficient memory I will try two things:
>Enable automatic heap dump on OUTOFMEMORY by adding JVM parameters
-20: Heapdumponoutofmemoryerror XX: heapdumppath = / tmp > try to connect to the JVM using jconsole to see if there is any abnormal mode