-
Bug
-
Resolution: Unresolved
-
Medium
-
None
since aprox 10/20/2017 we have seen our nitrogen and oxygen CSIT jobs fail
because the karaf process is using so much memory that the OS has to kill it:
the karaf console gets a message like this:
/tmp/karaf-0.8.0-SNAPSHOT/bin/karaf: line 422: 11539 Killed
${KARAF_EXEC} "${JAVA}" ${JAVA_OPTS} "$NON_BLOCKING_PRNG" -Djava.endorsed.dirs="${JAVA_ENDORSED_DIRS}" -Djava.ext.dirs="${JAVA_EXT_DIRS}" -Dkaraf.instances="${KARAF_HOME}/instances" -Dkaraf.home="${KARAF_HOME}" -Dkaraf.base="${KARAF_BASE}" -Dkaraf.data="${KARAF_DATA}" -Dkaraf.etc="${KARAF_ETC}" -Dkaraf.restart.jvm.supported=true -Djava.io.tmpdir="${KARAF_DATA}/tmp" -Djava.util.logging.config.file="${KARAF_BASE}/etc/java.util.logging.properties" ${KARAF_SYSTEM_OPTS} ${KARAF_OPTS} ${OPTS} "$@" -classpath "${CLASSPATH}" ${MAIN}
During a live debug session, we witnessed the java process consuming arpox 1.6G
of memory. (xmx is set to 2G and the host system has 4G). At some point, in a manner
of 5-10s the mem usage jumped up near 3.5G and the process was killed. It was very
sudden.
1.
|
Confirm ODL terminated by Kernel OOM killer |
|
Verified | Jamo Luhrsen |
2.
|
Try disabling Kernel OOM killer to see if we see more from JVM |
|
Resolved | Jamo Luhrsen |
3.
|
Track native memory usage via jcmd |
|
Resolved | Jamo Luhrsen |
4.
|
Try out LevelDB Java impl instead of JNI |
|
Verified | Stephen Kitt |
5.
|
Track heap & native (!) memory usage logs |
|
Resolved | Unassigned |
