https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/CONTROLLER-1693 controller <p>This manifests as a CSIT failure <span class="error">&#91;0&#93;</span>. The UnreachableMember is another issue (<a href="https://jira.opendaylight.org/browse/CONTROLLER-1645" title="shard moved during 1M bgp prefix advertizing (with tell-based=true)" class="issue-link" data-issue-key="CONTROLLER-1645">CONTROLLER-1645</a> for example). It is possible that cluster members end up with an inconsistent shard configuration.</p> <p>Karaf.log on member-1 <span class="error">&#91;1&#93;</span> shows the replica removal started at 01:55:43,472, then this happened:<br/> 2017-05-22 01:56:06,569 | WARN | lt-dispatcher-32 | aftActorLeadershipTransferCohort | 193 - org.opendaylight.controller.sal-akka-raft - 1.5.0.Carbon | member-1-shard-default-config: Failed to transfer leadership in 10.01 s<br/> 2017-05-22 01:56:06,572 | INFO | lt-dispatcher-22 | Shard | 192 - org.opendaylight.controller.sal-clustering-commons - 1.5.0.Carbon | Stopping Shard member-1-shard-default-config</p> <p>Finally, the test teardown started adding the replica back at 01:56:28,959.<br/> Thus even though the test was waiting 45 seconds, members only have 20 seconds to realize the previous leader is gone (we can add more time to the test if needed).</p> <p>As member-3 karaf.log <span class="error">&#91;2&#93;</span> shows no activity between 01:56:03,165 and 01:56:56,244 it looks like member-1 was perhaps somehow still a leader, but "has no leader" response <span class="error">&#91;3&#93;</span> from member-1 when adding the replica back proves there really was no leader, at least from member-1 point of view.</p> <p>Every member shows multiple UnreachableMember messages. Not sure if the subsequent ones are the cause or the result of missing the leader.</p> <p><span class="error">&#91;0&#93;</span> <a href="https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/715/archives/log.html.gz#s1-s36-t1-k2-k13-k1-k3-k1" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/715/archives/log.html.gz#s1-s36-t1-k2-k13-k1-k3-k1</a><br/> <span class="error">&#91;1&#93;</span> <a href="https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/715/archives/odl1_karaf.log.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/715/archives/odl1_karaf.log.gz</a><br/> <span class="error">&#91;2&#93;</span> <a href="https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/715/archives/odl3_karaf.log.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/715/archives/odl3_karaf.log.gz</a><br/> <span class="error">&#91;3&#93;</span> <a href="https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/715/archives/log.html.gz#s1-s36-t1-k2-k14-k2-k3-k1-k4-k7-k1" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/715/archives/log.html.gz#s1-s36-t1-k2-k14-k2-k3-k1-k4-k7-k1</a></p> <p>Operating System: All<br/> Platform: All</p> CONTROLLER-1693

UnreachableMember during remove-shard-replica prevents new leader to get elected

Bug Resolved Done Unassigned Vratko Polak Mon, 22 May 2017 08:47:46 +0000 Tue, 25 Jul 2023 08:24:40 +0000 Mon, 18 Sep 2017 12:12:37 +0000 clustering 0 2 <p>This seems to be a stable failure for listener tests when the listener is located on the leader member and shard replica is removed there. It happened two runs in a row, both for module-based and prefix-based shard tests.</p> <p>Example karaf.log <span class="error">&#91;4&#93;</span>: from 09:14:05,199 to 09:14:57,946.</p> <p><span class="error">&#91;4&#93;</span> <a href="https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/720/archives/odl2_karaf.log.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/720/archives/odl2_karaf.log.gz</a></p> <p>&gt; when the listener is located on the leader member and shard replica is removed there</p> <p>It also happened once <span class="error">&#91;5&#93;</span> when the (prefix-based shard) replica was removed from a leader on a different member than the listener.<br/> Previously, that scenario was running into <a href="https://jira.opendaylight.org/browse/CONTROLLER-1694" title="unsubscribe-dtcl fails with NullPointerException when UnreachableMember happens" class="issue-link" data-issue-key="CONTROLLER-1694"><del>CONTROLLER-1694</del></a> instead.</p> <p>&gt; Example karaf.log <span class="error">&#91;4&#93;</span></p> <p>From 09:29:43,066 to 09:31:27,574.</p> <p><span class="error">&#91;5&#93;</span> <a href="https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/720/archives/log.html.gz#s1-s38-t3-k2-k12-k1-k3-k1" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/720/archives/log.html.gz#s1-s38-t3-k2-k12-k1-k3-k1</a></p> <p>New debug logs <a href="https://jenkins.opendaylight.org/sandbox/job/controller-csit-3node-clustering-only-carbon-2nd/10/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/sandbox/job/controller-csit-3node-clustering-only-carbon-2nd/10/</a></p> <p>Odl used from <a href="https://jenkins.opendaylight.org/releng/view/integration/job/integration-multipatch-test-carbon/37/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/integration/job/integration-multipatch-test-carbon/37/</a></p> <p>New debug logs (including akka)<br/> <a href="https://jenkins.opendaylight.org/sandbox/job/controller-csit-3node-clustering-only-carbon-2nd/12/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/sandbox/job/controller-csit-3node-clustering-only-carbon-2nd/12/</a></p> <p>Odl was built from controller=63/57763/2:99/57699/3 at<br/> <a href="https://jenkins.opendaylight.org/releng/view/integration/job/integration-multipatch-test-carbon/43" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/integration/job/integration-multipatch-test-carbon/43</a></p> <p>New debug logs (including akka)<br/> <a href="https://jenkins.opendaylight.org/sandbox/job/controller-csit-3node-clustering-only-carbon-2nd/13/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/sandbox/job/controller-csit-3node-clustering-only-carbon-2nd/13/</a></p> <p>Odl built from controller=70/57770/4:99/57699/3 at<br/> <a href="https://jenkins.opendaylight.org/releng/view/integration/job/integration-multipatch-test-carbon/45/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/integration/job/integration-multipatch-test-carbon/45/</a></p> <p>Shard without the leader<br/> <a href="https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-only-carbon-2nd/13/archives/log.html.gz#s1-s2-t1-k2-k13-k1-k3-k1" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-only-carbon-2nd/13/archives/log.html.gz#s1-s2-t1-k2-k13-k1-k3-k1</a></p> <p>Testing without akka logs:<br/> ----------------------------<br/> Odl built from: controller=22/57822/1<br/> <a href="https://jenkins.opendaylight.org/releng/view/integration/job/integration-multipatch-test-carbon/49/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/integration/job/integration-multipatch-test-carbon/49/</a></p> <p>Testing job: <a href="https://jenkins.opendaylight.org/sandbox/job/controller-csit-3node-clustering-only-carbon-2nd/21/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/sandbox/job/controller-csit-3node-clustering-only-carbon-2nd/21/</a></p> <p>Link to problem:<a href="https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-only-carbon-2nd/21/archives/log.html.gz#s1-s2-t1-k2-k13-k1-k3-k1" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-only-carbon-2nd/21/archives/log.html.gz#s1-s2-t1-k2-k13-k1-k3-k1</a></p> <p>Testing with akka logs:<br/> ----------------------------<br/> Odl built from: controller=22/57822/1:99/57699/5<br/> <a href="https://jenkins.opendaylight.org/releng/view/integration/job/integration-multipatch-test-carbon/48/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/integration/job/integration-multipatch-test-carbon/48/</a></p> <p>Testing job: <a href="https://jenkins.opendaylight.org/sandbox/job/controller-csit-3node-clustering-only-carbon-2nd/20/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/sandbox/job/controller-csit-3node-clustering-only-carbon-2nd/20/</a></p> <p>Link to problem:<a href="https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-only-carbon-2nd/20/archives/log.html.gz#s1-s2-t1-k2-k13-k1-k3-k1" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-only-carbon-2nd/20/archives/log.html.gz#s1-s2-t1-k2-k13-k1-k3-k1</a></p> <p>This is still present on Releng <span class="error">&#91;19&#93;</span>.</p> <p><span class="error">&#91;19&#93;</span> <a href="https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/736/log.html.gz#s1-s38-t3-k2-k12-k1-k3-k1" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/736/log.html.gz#s1-s38-t3-k2-k12-k1-k3-k1</a></p> <p>Marking <a href="https://jira.opendaylight.org/browse/CONTROLLER-1706" title="Large transaction traffic prevents leader to be moved" class="issue-link" data-issue-key="CONTROLLER-1706"><del>CONTROLLER-1706</del></a> as dependency.</p> <p>If the new election takes too long, well-timed UnreachableMember could actually make the election finish sooner.<br/> So if the suite still fails, we should be fixing <a href="https://jira.opendaylight.org/browse/CONTROLLER-1706" title="Large transaction traffic prevents leader to be moved" class="issue-link" data-issue-key="CONTROLLER-1706"><del>CONTROLLER-1706</del></a> instead of this.</p> <p>Also, it will save me some time, as now I can assign failures to <a href="https://jira.opendaylight.org/browse/CONTROLLER-1706" title="Large transaction traffic prevents leader to be moved" class="issue-link" data-issue-key="CONTROLLER-1706"><del>CONTROLLER-1706</del></a> without karaf.log investigation.</p> <p>Keeping this open for now, as it is possible that Unreachablemember could mess with elections even after <a href="https://jira.opendaylight.org/browse/CONTROLLER-1706" title="Large transaction traffic prevents leader to be moved" class="issue-link" data-issue-key="CONTROLLER-1706"><del>CONTROLLER-1706</del></a> is fixed.</p> <p>&gt; Keeping this open for now, as it is possible that Unreachablemember<br/> &gt; could mess with elections even after <a href="https://jira.opendaylight.org/browse/CONTROLLER-1706" title="Large transaction traffic prevents leader to be moved" class="issue-link" data-issue-key="CONTROLLER-1706"><del>CONTROLLER-1706</del></a> is fixed.</p> <p>No such failures were seen, marking as fixed.</p> Blocks CONTROLLER-1706 Development External issue ID 8524 External issue URL Issue Type Rank 0|i02s5b: