https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/CONTROLLER-1706 controller <p>This symptom is affecting current cluster testing, but after fixing other bugs this might no longer be critical. This has been only seen on Sandbox so far.</p> <p>The robot symptom "Leader not found" <span class="error">&#91;0&#93;</span> is similar to <a href="https://jira.opendaylight.org/browse/CONTROLLER-1693" title="UnreachableMember during remove-shard-replica prevents new leader to get elected" class="issue-link" data-issue-key="CONTROLLER-1693"><del>CONTROLLER-1693</del></a> but this time there is no "UnreachableMember" seen in karaf.log <span class="error">&#91;1&#93;</span> (perhaps because <a href="https://jira.opendaylight.org/browse/CONTROLLER-1703" title="Tweak Akka and Java timeouts to a reasonable compromise between stability and failure detection" class="issue-link" data-issue-key="CONTROLLER-1703">CONTROLLER-1703</a>).<br/> <a href="https://jira.opendaylight.org/browse/CONTROLLER-1675" title="Leadership transfer failed: Follower is not ready to become leader" class="issue-link" data-issue-key="CONTROLLER-1675">CONTROLLER-1675</a> is similar, but here the leader moving call (remove-shard-replica in this case) does not fail, and karaf.log just states the leadership transfer was not successful:</p> <p>2017-06-06 07:51:36,160 | WARN | lt-dispatcher-28 | aftActorLeadershipTransferCohort | 193 - org.opendaylight.controller.sal-akka-raft - 1.5.1.SNAPSHOT | member-1-shard-default-config: Failed to transfer leadership in 10.01 s</p> <p>I suspect the large transaction is the creation of a large list at the start of write-transactions, see <a href="https://jira.opendaylight.org/browse/CONTROLLER-1703" title="Tweak Akka and Java timeouts to a reasonable compromise between stability and failure detection" class="issue-link" data-issue-key="CONTROLLER-1703">CONTROLLER-1703</a>. But I am not sure whether the following proves or disproves that:</p> <p>2017-06-06 07:52:40,474 | INFO | ternal.Finalizer | lientBackedTransaction$Finalizer | 199 - org.opendaylight.controller.sal-distributed-datastore - 1.5.1.SNAPSHOT | Aborted orphan transaction ClientSnapshot</p> {identifier=member-1-datastore-config-fe-0-txn-7-0} <p><span class="error">&#91;0&#93;</span> <a href="https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-only-carbon/4/log.html.gz#s1-s18-t1-k2-k12-k1-k3-k1" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-only-carbon/4/log.html.gz#s1-s18-t1-k2-k12-k1-k3-k1</a><br/> <span class="error">&#91;1&#93;</span> <a href="https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-only-carbon/4/odl1_karaf.log.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-only-carbon/4/odl1_karaf.log.gz</a></p> <p>Operating System: All<br/> Platform: All</p> CONTROLLER-1706

Large transaction traffic prevents leader to be moved

Bug Resolved Done Jakub Morvay Vratko Polak Tue, 6 Jun 2017 12:49:32 +0000 Tue, 25 Jul 2023 08:24:41 +0000 Wed, 21 Jun 2017 09:12:20 +0000 clustering 0 3 <p>Note that this happens both for module-based and prefix-based shards (tell-based protocol) but only when 3 writers/producers (and a single listener) are used. Similar suite with single writer/producer passes (mostly).</p> <p>With the initial big transaction removed <span class="error">&#91;2&#93;</span> (now merged to stable/carbon), this failure still affects remove-shard-replica when there are frequent small transactions, but only for module-based shards (not prefix-based ones) for some reason.</p> <p>Recent sandbox, robot <span class="error">&#91;3&#93;</span>, member-1 karaf.log (huge) <span class="error">&#91;4&#93;</span>:</p> <p>2017-06-06 19:14:27,516 | WARN | lt-dispatcher-25 | aftActorLeadershipTransferCohort | 193 - org.opendaylight.controller.sal-akka-raft - 1.5.1.SNAPSHOT | member-1-shard-default-config: Failed to transfer leadership in 10.03 s</p> <p><span class="error">&#91;2&#93;</span> <a href="https://git.opendaylight.org/gerrit/58355" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/58355</a><br/> <span class="error">&#91;3&#93;</span> <a href="https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-only-carbon/6/log.html.gz#s1-s18-t1-k2-k12-k1-k3-k1" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-only-carbon/6/log.html.gz#s1-s18-t1-k2-k12-k1-k3-k1</a><br/> <span class="error">&#91;4&#93;</span> <a href="https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-only-carbon/6/odl1_karaf.log.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-only-carbon/6/odl1_karaf.log.gz</a></p> <p>&gt; (not prefix-based ones) for some reason.</p> <p>For comparison, prefix shard performance:</p> <p>2017-06-06 19:23:12,676 | INFO | ult-dispatcher-3 | aftActorLeadershipTransferCohort | 193 - org.opendaylight.controller.sal-akka-raft - 1.5.1.SNAPSHOT | member-3-shard-id-ints!-config: Successfully transferred leadership to null in 2.031 s</p> <p>I have analyzed logs from run <span class="error">&#91;0&#93;</span>. The only differences between this job and the jobs mentioned above by Vratko are that we are using PoisonPill to kill shard leader replica (not remove-shard-replica rpc) and we have debug logs for raft messages.</p> <p>Moreover, I have been looking on Clean Leader Shutdown suite, not on Remote Listener suite mentioned by Vratko but I believe they are failing because of the same reason.</p> <p>Scenario goes as follows. We have three node cluster and on leader, we start writing data to shard-default-config shard (that's member-3 in this particular run). Then we kill shard-default-config shard on leader node (we send PoisonPill to member-3-shard-default-config shard). We expect that followers will elect new leader quickly, in span of small amount of election timeouts. But this is not the case. Leader is not elected in one minute and test fails.</p> <p>From the logs we can see the leader shard is going down in 11:06:31,494<br/> 2017-06-08 11:06:31,494 | INFO | lt-dispatcher-20 | Shard | 192 - org.opendaylight.controller.sal-clustering-commons - 1.5.1.SNAPSHOT | Stopping Shard member-3-shard-default-config<br/> 2017-06-08 11:06:31,494 | DEBUG | lt-dispatcher-20 | Shard | 192 - org.opendaylight.controller.sal-clustering-commons - 1.5.1.SNAPSHOT | member-3-shard-default-config: Aborting 307 pending queued transactions</p> <p>When we look on followers, they are still processing AppendEntries messages for shard-default-config shard from leader for some small amount of time. But we can see they timeout elections properly, roughly after 10 seconds the leader is down. We can see this on member-2</p> <p>2017-06-08 11:06:41,479 | TRACE | lt-dispatcher-30 | Shard | 192 - org.opendaylight.controller.sal-clustering-commons - 1.5.1.SNAPSHOT | Received message ElectionTimeout<br/> 2017-06-08 11:06:41,479 | DEBUG | lt-dispatcher-30 | Shard | 192 - org.opendaylight.controller.sal-clustering-commons - 1.5.1.SNAPSHOT | member-2-shard-default-config (Follower): Checking for leader akka.tcp://opendaylight-cluster-data@10.29.15.136:2550 in the cluster unreachable set []<br/> 2017-06-08 11:06:41,481 | DEBUG | lt-dispatcher-30 | Shard | 192 - org.opendaylight.controller.sal-clustering-commons - 1.5.1.SNAPSHOT | member-2-shard-default-config (Follower): Leader akka.tcp://opendaylight-cluster-data@10.29.15.136:2550 cluster status is Up - leader is available<br/> 2017-06-08 11:06:41,481 | DEBUG | lt-dispatcher-30 | Shard | 192 - org.opendaylight.controller.sal-clustering-commons - 1.5.1.SNAPSHOT | member-2-shard-default-config (Follower): Received ElectionTimeout but leader appears to be available</p> <p>And this is repeated every 10 seconds, until the test ends with failure. We can see same logs also on member-1.</p> <p>When we look closely into the implementation of handling election timeout on followers, we can see that followers are checking also the cluster status of leader before switching to candidate. And they do it for 180 seconds (if the election timeout is 10 seconds). This was introduced in <a href="https://git.opendaylight.org/gerrit/#/c/43265/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/43265/</a> as a fix for <a href="https://jira.opendaylight.org/browse/CONTROLLER-1495" title="Prevent Follower from becoming Candidate when Akka Cluster reports Leader as Reachable" class="issue-link" data-issue-key="CONTROLLER-1495"><del>CONTROLLER-1495</del></a>.</p> <p>Clearly, this prevents quick elections in the case the leader shard actor is down but its cluster node is still up and the leadership transfer was not successful. For example, this is the case if you send PoisonPill to leader shard actor or if RaftActor's pauseLeader <span class="error">&#91;1&#93;</span> method fails during leadership transfer attempt after sending Shutdown message to leader shard actor (this can happen as a part of remove-shard-replica rpc operation).</p> <p><span class="error">&#91;0&#93;</span> <a href="https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-only-carbon-2nd/17/log.html.gz#s1-s2" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-only-carbon-2nd/17/log.html.gz#s1-s2</a><br/> <span class="error">&#91;1&#93;</span> <a href="https://github.com/opendaylight/controller/blob/87551f3a44856d7494ffef678b63bb01a1b4ab2e/opendaylight/md-sal/sal-akka-raft/src/main/java/org/opendaylight/controller/cluster/raft/RaftActorLeadershipTransferCohort.java#L103" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/opendaylight/controller/blob/87551f3a44856d7494ffef678b63bb01a1b4ab2e/opendaylight/md-sal/sal-akka-raft/src/main/java/org/opendaylight/controller/cluster/raft/RaftActorLeadershipTransferCohort.java#L103</a></p> <p>yes - I warned about side-effects in the patch with just blindly killing the shard in the tests with PoisonPill. On election timeout, followers check akka's cluster state - this is to avoid spurious elections if the leader shard is busy for period of time. We want to avoid deposing the leader unless we can be reasonably sure the node is not available. Normally in production the leader node either is shut down or becomes unavailable due to network partition. In either case, akka's cluster state will mark the node as unreachable/down so followers will initiate a new election on timeout. On graceful leader shutdown, we try to transfer leadership to speed up the process. </p> <p>We do have an upper deadline to initiate a new election on election timeout as a fail-safe as you mentioned. This would occur if the leader shard actor had a hard failure and couldn't recover or the process is hosed (low memory, out of memory etc), either of which should be uncommon. </p> <p>In a 3-node setup, what are you really trying to test by killing the leader shard actor? Why don't you just shut the leader node down, or kill the process, or remove the shard replica? If you really want stop the shard while keeping the process up then I would suggest stopping it gracefully so it transfers leadership.</p> <p>(In reply to Jakub Morvay from comment #4)</p> <p>&gt; Clearly, this prevents quick elections in the case the leader shard actor is<br/> &gt; down but its cluster node is still up and the leadership transfer was not<br/> &gt; successful. For example, this is the case if you send PoisonPill to leader<br/> &gt; shard actor or if RaftActor's pauseLeader <span class="error">&#91;1&#93;</span> method fails during leadership<br/> &gt; transfer attempt after sending Shutdown message to leader shard actor (this<br/> &gt; can happen as a part of remove-shard-replica rpc operation).<br/> &gt; </p> <p>yes - but a failure in pauseLeader or leadership transfer indicates follower unavailability (normally at least), either followers are down or the leader is partitioned. For the latter, the followers would detect the partition as well and initiate a new election.</p> <p>(In reply to Tom Pantelis from comment #5)<br/> &gt; In a 3-node setup, what are you really trying to test by killing the leader<br/> &gt; shard actor? Why don't you just shut the leader node down, or kill the<br/> &gt; process, or remove the shard replica? If you really want stop the shard<br/> &gt; while keeping the process up then I would suggest stopping it gracefully so<br/> &gt; it transfers leadership.</p> <p>We have used remove-shard-replica rpc for this and it is still used on releng. We are going to try to stop shard gracefully in our tests, I'm currently working on such rpcs. However I believe this failure will be still present, because pauseLeader method will fail.</p> <p>(In reply to Tom Pantelis from comment #6)<br/> &gt; yes - but a failure in pauseLeader or leadership transfer indicates follower<br/> &gt; unavailability (normally at least), either followers are down or the leader<br/> &gt; is partitioned. For the latter, the followers would detect the partition as<br/> &gt; well and initiate a new election.</p> <p>Well, is that true for pauseLeader method "timeout"? If leader cannot complete all pending tasks in election timeout period, it just aborts the leadership transfer and do nothing (apart from shutting itself, become non-voting etc.). If we know, that leader is stepping down anyway, shouldn't we send TimeoutNow message to random follower to speed things up and not wait 180 seconds for new leader to emerge?</p> <p>(In reply to Jakub Morvay from comment #7)<br/> &gt; (In reply to Tom Pantelis from comment #5)<br/> &gt; &gt; In a 3-node setup, what are you really trying to test by killing the leader<br/> &gt; &gt; shard actor? Why don't you just shut the leader node down, or kill the<br/> &gt; &gt; process, or remove the shard replica? If you really want stop the shard<br/> &gt; &gt; while keeping the process up then I would suggest stopping it gracefully so<br/> &gt; &gt; it transfers leadership.<br/> &gt; <br/> &gt; We have used remove-shard-replica rpc for this and it is still used on<br/> &gt; releng. We are going to try to stop shard gracefully in our tests, I'm<br/> &gt; currently working on such rpcs. However I believe this failure will be still<br/> &gt; present, because pauseLeader method will fail.</p> <p>pauseLeader in Shard tries to wait for all pending transactions to complete within the election timeout interval. Why would you to expect this to time out in the test suite? </p> <p>In any event, if you anticipate leadership transfer could fail then you can increase the expected deadline in the suite to take into account the potential 180 second timeout.</p> <p>(In reply to Tom Pantelis from comment #9) <br/> &gt; pauseLeader in Shard tries to wait for all pending transactions to complete<br/> &gt; within the election timeout interval. Why would you to expect this to time<br/> &gt; out in the test suite? <br/> &gt; <br/> &gt; In any event, if you anticipate leadership transfer could fail then you can<br/> &gt; increase the expected deadline in the suite to take into account the<br/> &gt; potential 180 second timeout.</p> <p>Yeah well, we have seen such case, where leader couldn't complete all txs within the election timeout interval.</p> <p>(In reply to Jakub Morvay from comment #8)<br/> &gt; (In reply to Tom Pantelis from comment #6)<br/> &gt; &gt; yes - but a failure in pauseLeader or leadership transfer indicates follower<br/> &gt; &gt; unavailability (normally at least), either followers are down or the leader<br/> &gt; &gt; is partitioned. For the latter, the followers would detect the partition as<br/> &gt; &gt; well and initiate a new election.<br/> &gt; <br/> &gt; Well, is that true for pauseLeader method "timeout"? If leader cannot<br/> &gt; complete all pending tasks in election timeout period, it just aborts the<br/> &gt; leadership transfer and do nothing (apart from shutting itself, become<br/> &gt; non-voting etc.). If we know, that leader is stepping down anyway, shouldn't<br/> &gt; we send TimeoutNow message to random follower to speed things up and not<br/> &gt; wait 180 seconds for new leader to emerge?</p> <p>Keep in mind that hitting the 180 second deadline should be very uncommon. You hit it b/c you blindly killed the leader actor ungracefully while keeping the process up which is abnormal.</p> <p>If pauseLeader fails it most likely means it can't get consensus for a transaction in which case leadership transfer would fail as well as there aren't enough available nodes to elect a new leader. Of course it could time out for uncommon reasons, eg akka's transport is backed up or followers are too busy or in a bad state or the volume of transactions is large.</p> <p>Just sending TimeoutNow message to a random follower isn't ideal as the follower may not be able to collect the votes to become leader. This is why the leader first attempts to "catch up" at least one follower. </p> <p>If leadership transfer times out, could you just send TimeoutNow to all followers? We could but there could be unwanted side-effects. In the common case where a majority of followers isn't available, sending TimeoutNow probably won't help any. However the TimeoutNow message may get queued in akka's transport layer and eventually be delivered at some point which may not be desirable. Same could happen for the uncommon reasons above.</p> <p>(In reply to Jakub Morvay from comment #10)<br/> &gt; (In reply to Tom Pantelis from comment #9) <br/> &gt; &gt; pauseLeader in Shard tries to wait for all pending transactions to complete<br/> &gt; &gt; within the election timeout interval. Why would you to expect this to time<br/> &gt; &gt; out in the test suite? <br/> &gt; &gt; <br/> &gt; &gt; In any event, if you anticipate leadership transfer could fail then you can<br/> &gt; &gt; increase the expected deadline in the suite to take into account the<br/> &gt; &gt; potential 180 second timeout.<br/> &gt; <br/> &gt; Yeah well, we have seen such case, where leader couldn't complete all txs<br/> &gt; within the election timeout interval.</p> <p>When you're testing scale you usually have to be more lenient with the test expectations and deadlines. The test VM environment could randomly cause delays wrt to system and network resources and result in intermittent failures. Even if you forced a TimeoutNow message, if there's a large volume of pending transactions then the follower's queue will likely be backed up so no guarantee when the TimeoutNow message would be delivered.</p> <p>Thinking about this some more, if pauseLeader times out, I think it does make sense to continue with leadership transfer instead of aborting. The shard may have a lot of transactions queued up which it can't finish in time but there may still be a follower that is caught up (ie whose matchIndex equals the leader's lastIndex) or would be caught up if leadership transfer continued. Worst case is no follower is available and the "catch up" phase of leadership transfer also times out which would lengthen shut down time but that should be fine. I can look into that.</p> <p>As a last resort if leadership transfer times out, we could force the TimeoutNow message to "active" followers, ie those that have responded recently so should be up.</p> <p>"Failed to transfer leadership" still happens occasionally (<span class="error">&#91;2&#93;</span> at 09:14:29,865). For example after remove-shard-replica with single transaction-writer <span class="error">&#91;3&#93;</span> (on a node separate to leader of module-based shard, tell-based protocol).</p> <p><span class="error">&#91;2&#93;</span> <a href="https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/740/odl1_karaf.log.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/740/odl1_karaf.log.gz</a><br/> <span class="error">&#91;3&#93;</span> <a href="https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/740/log.html.gz#s1-s20-t3-k2-k9" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/740/log.html.gz#s1-s20-t3-k2-k9</a></p> <p>Also the original "Leader not found" symptom is seen <span class="error">&#91;4&#93;</span> occasionally.</p> <p><span class="error">&#91;4&#93;</span> <a href="https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/740/log.html.gz#s1-s36-t1-k2-k12-k1-k3-k1" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/740/log.html.gz#s1-s36-t1-k2-k12-k1-k3-k1</a></p> <p>(In reply to Vratko Polák from comment #14)<br/> &gt; "Failed to transfer leadership" still happens occasionally (<span class="error">&#91;2&#93;</span> at<br/> &gt; 09:14:29,865). For example after remove-shard-replica with single<br/> &gt; transaction-writer <span class="error">&#91;3&#93;</span> (on a node separate to leader of module-based shard,<br/> &gt; tell-based protocol).<br/> &gt; <br/> &gt; <span class="error">&#91;2&#93;</span><br/> &gt; <a href="https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-</a><br/> &gt; clustering-only-carbon/740/odl1_karaf.log.gz<br/> &gt; <span class="error">&#91;3&#93;</span><br/> &gt; <a href="https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-</a><br/> &gt; clustering-only-carbon/740/log.html.gz#s1-s20-t3-k2-k9</p> <p>No matter what we do, leadership transfer is a best effort and may not be able to successfully complete in time under load for the reasons I've mentioned. You have to take that into account in the tests with more lenient deadlines.</p> <p>Submitted <a href="https://git.opendaylight.org/gerrit/#/c/58740/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/58740/</a></p> Blocks CONTROLLER-1693 CONTROLLER-1703 Development External issue ID 8606 External issue URL Issue Type Rank 0|i02s87: