https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/CONTROLLER-1762 controller <p>Description of problem: On running longevity tests in a clustered ODL setup we see that one of the ODL instances seems to be up and running as reported by ps output, systemctl and netstat listening ports, however it doesn't seem to be functional. We could not even ssh into the karaf terminal using ssh -p 8101 karaf@172.16.0.16 until we restarted opendaylight. On performing a service restart we were able to get into the karaf shell and ODL seemed to come back up.<br/> Out of the other two instances of ODL, one was killed due to OOM and the other seemed to be running fine. This happens after about 42 hours of running the tests.<br/> Setup:<br/> 3 ODLs<br/> 3 OpenStack Controllers<br/> 3 Compute nodes</p> <p>Test:<br/> Create 40 neutron resources (rotuers, networks etc) 2 at a time using Rally and delete them over and over again. This is a long running low stress test.</p> <p>Entire Karaf Log: <a href="http://8.43.86.1:8088/smalleni/karaf-controller-0.log.tar.gz" class="external-link" target="_blank" rel="nofollow noopener">http://8.43.86.1:8088/smalleni/karaf-controller-0.log.tar.gz</a></p> <p>ODL RPM from upstream: python-networking-odl-11.0.0-0.20170806093629.2e78dca.el7ost.noarch</p> <p>Operating System: All<br/> Platform: All</p> CONTROLLER-1762

ODL is up and ports are listening but not functional

Bug Resolved Duplicate Unassigned Sai Sindhur Malleni Mon, 28 Aug 2017 21:25:12 +0000 Thu, 19 Oct 2017 21:25:01 +0000 Tue, 12 Sep 2017 17:44:14 +0000 Carbon clustering 0 6 <p>ODL became non-functional around 10:44 UTC 08/28/2017. This was confirmed as collectd which talks tothe Karaf JMX suddenly stopped reporting values for heap size. Collectd was able to talk to Karaf JMX after the service restart. The break can be clearly observed at: <a href="https://snapshot.raintank.io/dashboard/snapshot/nf6OWq7jNSeT6vwjM71jlUSWc31E9LdW" class="external-link" target="_blank" rel="nofollow noopener">https://snapshot.raintank.io/dashboard/snapshot/nf6OWq7jNSeT6vwjM71jlUSWc31E9LdW</a></p> <p>the karaf.log file is ~1G so it's hard to debug with it. I did see a ton of timed out messages in genius.lockmanager-impl at one point, but that could just be a <br/> symptom. And in reality they are taking up less than 1% of the total log messages.</p> <p>&gt; it doesn't seem to be functional. We could not even ssh<br/> &gt; into the karaf terminal using ssh -p 8101 karaf@172.16.0.16</p> <p>Sai, when you hit this kind of situation, it would interesting if we could have a "thread stack dump" of that JVM, to see if we can spot anything obvious (like e.g. an obvious deadlock, or an extreme number of threads). Use the JDK's "jstack" utility to obtain this. Try "jstack -h" to learn it, if you've never used it. Use the -l flag. You'll need to check as what user you have to run jstack to work - try first on a non-stuck instance? If jstack still doesn't work, its -F flag helps sometimes, but having to use that is a bad sign. If still doesn't work, then we would probably need help from our openjdk team friends at Red Hat to be able to understand what horrible thing ODL code may be doing to get a JVM that badly stuck.</p> <p>Sure, Michael, Next time I will do that.</p> <p>If it helps: The karaf thread count <a href="https://snapshot.raintank.io/dashboard/snapshot/EgrJsRB7HJ6tl1pjLlSY4hb6wWvJS7nT" class="external-link" target="_blank" rel="nofollow noopener">https://snapshot.raintank.io/dashboard/snapshot/EgrJsRB7HJ6tl1pjLlSY4hb6wWvJS7nT</a></p> <p>We can see that arund 10:44 UTC the thread count suddenly spikes and falls back after a restart.</p> <p>ODL RPM used was opendaylight-6.2.0-0.1.20170817rel1931.el7.noarch</p> <p>&gt; If it helps: The karaf thread count </p> <p>Sai, that actually is an interesting observation - it's certainly possible that, in addition to OOM problems due to Memory Leaks related to TransactionChain (<a href="https://jira.opendaylight.org/browse/CONTROLLER-1756" title="OOM due to huge Map in ShardDataTree" class="issue-link" data-issue-key="CONTROLLER-1756"><del>CONTROLLER-1756</del></a>) and broken clustering (<a href="https://jira.opendaylight.org/browse/CONTROLLER-1763" title="On restarting ODL on one node, ODL on another node dies in a clustered setup" class="issue-link" data-issue-key="CONTROLLER-1763"><del>CONTROLLER-1763</del></a>) we also have a "Thread leak" issue (i.e. unbounded creation of new threads, instead of correctly using a pool / Executor thing), somewhere in the code... could be anywhere - netvirt, genius, mdsal - who knows. But before digging more into that, we would first need definitive proof that is the underlying root cause here - maybe the systemctl status in <a href="https://jira.opendaylight.org/browse/CONTROLLER-1763" title="On restarting ODL on one node, ODL on another node dies in a clustered setup" class="issue-link" data-issue-key="CONTROLLER-1763"><del>CONTROLLER-1763</del></a> will give us that, let's see.</p> <p>Wondering if <a href="https://jira.opendaylight.org/browse/CONTROLLER-1755" title="RaftActor lastApplied index moves backwards" class="issue-link" data-issue-key="CONTROLLER-1755"><del>CONTROLLER-1755</del></a> may helped fix this - let's re-test and confirm is still seen.</p> Blocks CONTROLLER-1755 Duplicate CONTROLLER-1756 Development External issue ID 9063 External issue URL ODL SR Target Milestone Priority Rank 0|i02skn: