Operating System: All
Platform: All

ODL became non-functional around 10:44 UTC 08/28/2017. This was confirmed as collectd which talks tothe Karaf JMX suddenly stopped reporting values for heap size. Collectd was able to talk to Karaf JMX after the service restart. The break can be clearly observed at: https://snapshot.raintank.io/dashboard/snapshot/nf6OWq7jNSeT6vwjM71jlUSWc31E9LdW

the karaf.log file is ~1G so it's hard to debug with it. I did see a ton of timed out messages in genius.lockmanager-impl at one point, but that could just be a
symptom. And in reality they are taking up less than 1% of the total log messages.

> it doesn't seem to be functional. We could not even ssh
> into the karaf terminal using ssh -p 8101 karaf@172.16.0.16

Sai, when you hit this kind of situation, it would interesting if we could have a "thread stack dump" of that JVM, to see if we can spot anything obvious (like e.g. an obvious deadlock, or an extreme number of threads). Use the JDK's "jstack" utility to obtain this. Try "jstack -h" to learn it, if you've never used it. Use the -l flag. You'll need to check as what user you have to run jstack to work - try first on a non-stuck instance? If jstack still doesn't work, its -F flag helps sometimes, but having to use that is a bad sign. If still doesn't work, then we would probably need help from our openjdk team friends at Red Hat to be able to understand what horrible thing ODL code may be doing to get a JVM that badly stuck.

Sure, Michael, Next time I will do that.

If it helps: The karaf thread count https://snapshot.raintank.io/dashboard/snapshot/EgrJsRB7HJ6tl1pjLlSY4hb6wWvJS7nT

We can see that arund 10:44 UTC the thread count suddenly spikes and falls back after a restart.

ODL RPM used was opendaylight-6.2.0-0.1.20170817rel1931.el7.noarch

> If it helps: The karaf thread count

Sai, that actually is an interesting observation - it's certainly possible that, in addition to OOM problems due to Memory Leaks related to TransactionChain (CONTROLLER-1756) and broken clustering (CONTROLLER-1763) we also have a "Thread leak" issue (i.e. unbounded creation of new threads, instead of correctly using a pool / Executor thing), somewhere in the code... could be anywhere - netvirt, genius, mdsal - who knows. But before digging more into that, we would first need definitive proof that is the underlying root cause here - maybe the systemctl status in CONTROLLER-1763 will give us that, let's see.

Wondering if CONTROLLER-1755 may helped fix this - let's re-test and confirm is still seen.