https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/CONTROLLER-1738 controller <p>This is similar to <a href="https://jira.opendaylight.org/browse/CONTROLLER-1717" title="RequestTimeoutException due to &quot;Failed to transfer leadership&quot; after become-prefix-leader with RoleChangeNotification not delivered" class="issue-link" data-issue-key="CONTROLLER-1717"><del>CONTROLLER-1717</del></a> but a different message has been lost here.<br/> I believe <a href="https://jira.opendaylight.org/browse/CONTROLLER-1714" title="RequestTimeoutException after remove-shard-replica (without apparent cause)" class="issue-link" data-issue-key="CONTROLLER-1714"><del>CONTROLLER-1714</del></a> was exactly this, but with less evidence in karaf log.</p> <p>The Robot failure <span class="error">&#91;0&#93;</span> is the usual 120s timeout we can see caused by multiple bugs (from transaction writer for module-based shard, tell-based protocol):<br/> RequestTimeoutException: Timed out after 120.029805238seconds</p> <p>Looking at karaf log <span class="error">&#91;1&#93;</span> of member-1 (writer, old leader), we can see leadership has been successfully transferred at 04:04:59,250 but the information about the new leader being there has been lost:<br/> 2017-07-04 04:04:59,252 | INFO | lt-dispatcher-42 | LocalActorRef | 174 - com.typesafe.akka.slf4j - 2.4.18 | Message <span class="error">&#91;org.opendaylight.controller.cluster.datastore.messages.ShardLeaderStateChanged&#93;</span> from Actor<a href="#145361760" target="_blank" rel="noopener">akka://opendaylight-cluster-data/user/shardmanager-config/member-1-shard-default-config#145361760</a> to Actor<a href="#-591265397" target="_blank" rel="noopener">akka://opendaylight-cluster-data/user/shardmanager-config/member-1-shard-default-config/member-1-shard-default-config-notifier#-591265397</a> was not delivered. <span class="error">&#91;5&#93;</span> dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.</p> <p>So the member new it is a Follower, but it was unable to tell client who the new leader is.<br/> 2017-07-04 04:05:19,248 | WARN | monPool-worker-2 | AbstractShardBackendResolver | 199 - org.opendaylight.controller.sal-distributed-datastore - 1.5.1.Carbon | Failed to resolve shard<br/> java.util.concurrent.TimeoutException: Shard has no current leader</p> <p>Perhaps there is a common underlying Bug which causes occasional undelivered messages, and we see different symptoms depending on which message gets lost.</p> <p><span class="error">&#91;0&#93;</span> <a href="https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/771/log.html.gz#s1-s20-t1-k2-k8" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/771/log.html.gz#s1-s20-t1-k2-k8</a><br/> <span class="error">&#91;1&#93;</span> <a href="https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/771/odl1_karaf.log.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/771/odl1_karaf.log.gz</a></p> <p>Operating System: All<br/> Platform: All</p> CONTROLLER-1738

RequestTimeoutException due to "Shard has no current leader" after shutdown-shard-replica with ShardLeaderStateChanged not delivered

Bug Confirmed Unresolved Unassigned Vratko Polak Tue, 4 Jul 2017 08:53:03 +0000 Tue, 25 Jul 2023 08:24:44 +0000 clustering 0 3 <p>Can you describe the detailed sequence of the test? You mention shutdown-shard-replica - I assume this is the "backdoor" RPC to gracefully shut down a shard actor unbeknownst to the ShardManager. However the ShardManager does not know the shard actor was shutdown so still has a record of it. I believe the ShardLeaderStateChanged goes to dead letters b/c the notifier actor goes away with the shard actor, which wouldn't matter if the ShardManager had initiated the shutdown thru the normal code paths.</p> <p>I've warned about using that backdoor for testing - I still think it's better to use normal code paths.</p> <p>Right, since the notifier is a child of the shard actor, it gets reaped alongside with it. <a href="https://git.opendaylight.org/gerrit/59210" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/59210</a> adds a deathwatch, which should solve this case, I think.</p> <p>&gt; You mention shutdown-shard-replica</p> <p>Yes, it is a RPC implemented here <span class="error">&#91;2&#93;</span></p> <p>The interesting fact is that the test passes most of the time on RelEng.<br/> Here is the failure encountered on Sandbox run with debug logs <span class="error">&#91;3&#93;</span>.</p> <p><span class="error">&#91;2&#93;</span> <a href="https://git.opendaylight.org/gerrit/gitweb?p=controller.git;a=blob;f=opendaylight/md-sal/samples/clustering-test-app/provider/src/main/java/org/opendaylight/controller/clustering/it/provider/MdsalLowLevelTestProvider.java;h=e0e8d99d1aab3eac8df847bf7075de6e15b0257e;hb=refs/heads/stable/carbon#l574" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/gitweb?p=controller.git;a=blob;f=opendaylight/md-sal/samples/clustering-test-app/provider/src/main/java/org/opendaylight/controller/clustering/it/provider/MdsalLowLevelTestProvider.java;h=e0e8d99d1aab3eac8df847bf7075de6e15b0257e;hb=refs/heads/stable/carbon#l574</a><br/> <span class="error">&#91;3&#93;</span> <a href="https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-cls-only-carbon/1/log.html.gz#s1-s2-t1-k2-k8" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-cls-only-carbon/1/log.html.gz#s1-s2-t1-k2-k8</a></p> <p>As a workaround, we can change the tests <span class="error">&#91;4&#93;</span> to use remove-shard-replica instead.<br/> This bug would stay opened, but importance would be lower.</p> <p><span class="error">&#91;4&#93;</span> <a href="https://git.opendaylight.org/gerrit/60140" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/60140</a></p> <p><span class="error">&#91;4&#93;</span> merged, importance lowered.</p> Duplicate CONTROLLER-1714 Development External issue ID 8794 External issue URL Issue Type Rank 0|i02sfb: