https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/CONTROLLER-1754 controller <p>This is probably a duplicate of an already existing Bug.</p> <p><a href="https://jira.opendaylight.org/browse/CONTROLLER-1751" title="Sporadic cluster failure when member is restarted in OF cluster test" class="issue-link" data-issue-key="CONTROLLER-1751"><del>CONTROLLER-1751</del></a> reports very similar behavior for Nitrogen. Possibly the same bug, but the Robot symptom is different. While on Nitrogen Jolokia works, just reporting cluster is not in sync, on Carbon Jolokia does not find ShardManager <span class="error">&#91;0&#93;</span>.</p> <p><a href="https://jira.opendaylight.org/browse/NETCONF-454" title="Three node cluster does not recover from isolation+ rejoin of a member." class="issue-link" data-issue-key="NETCONF-454">NETCONF-454</a> is perhaps the same bug, but from its description that looks like a superset of this Bug. According to a comment <span class="error">&#91;1&#93;</span> there, this Bug can be fixed by <span class="error">&#91;2&#93;</span>, which is a fix for <a href="https://jira.opendaylight.org/browse/CONTROLLER-1627" title="LinkageError for union type after switchover or warm restart of instances cluster causing write tx to fail" class="issue-link" data-issue-key="CONTROLLER-1627"><del>CONTROLLER-1627</del></a>.</p> <p>This Bug is not easy to reproduce reliably, as single restart failure frequency is small. It only affects Netconf suite significantly, because there are multiple restarts.</p> <p><span class="error">&#91;0&#93;</span> <a href="https://logs.opendaylight.org/releng/jenkins092/netconf-csit-3node-clustering-only-carbon/630/log.html.gz#s1-s5-t13-k2-k2-k8-k1-k2-k1-k1-k3-k1" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/netconf-csit-3node-clustering-only-carbon/630/log.html.gz#s1-s5-t13-k2-k2-k8-k1-k2-k1-k1-k3-k1</a><br/> <span class="error">&#91;1&#93;</span> <a href="https://bugs.opendaylight.org/show_bug.cgi?id=8999#c7" class="external-link" target="_blank" rel="nofollow noopener">https://bugs.opendaylight.org/show_bug.cgi?id=8999#c7</a><br/> <span class="error">&#91;2&#93;</span> <a href="https://git.opendaylight.org/gerrit/60485" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/60485</a></p> <p>Operating System: All<br/> Platform: All</p> CONTROLLER-1754

Carbon: Sporadic cluster failure when member is restarted in Netconf cluster test

Bug Resolved Duplicate Unassigned Vratko Polak Tue, 22 Aug 2017 11:22:35 +0000 Thu, 19 Oct 2017 21:25:06 +0000 Sat, 9 Sep 2017 17:04:20 +0000 Nitrogen clustering 0 4 <p>Waiting for Sandbox test results to see whether the cherry-picked fix <span class="error">&#91;3&#93;</span> works.</p> <p><span class="error">&#91;3&#93;</span> <a href="https://git.opendaylight.org/gerrit/62148" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/62148</a></p> <p>&gt; cherry-picked fix <span class="error">&#91;3&#93;</span></p> <p>That does not work.</p> <p>I have also tried a change in test <span class="error">&#91;4&#93;</span> adding hard resets. It perhaps reduces the frequency, but does not prevent this failure, as Sandbox shows <span class="error">&#91;5&#93;</span>.</p> <p>I have run out of ideas.</p> <p><span class="error">&#91;4&#93;</span> <a href="https://git.opendaylight.org/gerrit/62194" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/62194</a><br/> <span class="error">&#91;5&#93;</span> <a href="https://logs.opendaylight.org/sandbox/jenkins091/netconf-csit-3node-clustering-only-carbon/20/log.html.gz#s1-s8-t19-k2-k2-k8-k1-k2-k1-k1-k3-k1" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/sandbox/jenkins091/netconf-csit-3node-clustering-only-carbon/20/log.html.gz#s1-s8-t19-k2-k2-k8-k1-k2-k1-k1-k3-k1</a></p> <p>Well, the karaf restarts don't do enough to clear state and end up being the equivalent of bundle reload.</p> <p>I think the correct fix is to either auto-detect, or expose as a knob, the mechanism of JVM shutdown:</p> <ul class="alternate" type="square"> <li>in karaf 4, signal restart as we do now</li> <li>everywhere else, including OSGi, terminate the VM</li> </ul> <p>It seems this bug is way less frequent on <del>all</del> (as opposed to <del>only</del>) jobs. I see only one failure <span class="error">&#91;6&#93;</span> and it is a 404 (as in jolokia not started) as opposed to the sync=false symptom before.</p> <p>This is important, because Releng/Builder decided <span class="error">&#91;7&#93;</span> to drop <del>only</del> jobs to speed up the distribution-test cycle.</p> <p><span class="error">&#91;6&#93;</span> <a href="https://logs.opendaylight.org/releng/jenkins092/netconf-csit-3node-clustering-all-carbon/385/log.html.gz#s1-s10-t13-k2-k2-k8-k1-k2-k1-k1-k2-k1-k4-k1" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/netconf-csit-3node-clustering-all-carbon/385/log.html.gz#s1-s10-t13-k2-k2-k8-k1-k2-k1-k1-k2-k1-k4-k1</a><br/> <span class="error">&#91;7&#93;</span> <a href="https://git.opendaylight.org/gerrit/62297" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/62297</a></p> <p>&gt;&gt; adding hard resets</p> <p>&gt; karaf restarts don't do enough to clear state</p> <p>The suite at this segment does:<br/> 0. kill JVM<br/> 1. delete data/ snapshot/ journal/ and similar. Basically only etc/ and karaf.log survive (although there may be obscure places left, where ODL applications can store their data).<br/> 2. start karaf again<br/> 3. wait for sync (when not synced in 5 minutes, it is this Bug).</p> Duplicate CONTROLLER-1751 Development External issue ID 9027 External issue URL Issue Type Rank 0|i02siv: