https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/CONTROLLER-1713 controller <p>Seen <span class="error">&#91;0&#93;</span> on RelEng first time with module-based shard (tell-based protocol).<br/> Member-3 was the old leader, member-2 became new leader.</p> <p>This is similar to <a href="https://jira.opendaylight.org/browse/CONTROLLER-1693" title="UnreachableMember during remove-shard-replica prevents new leader to get elected" class="issue-link" data-issue-key="CONTROLLER-1693"><del>CONTROLLER-1693</del></a> in that member-2 has seen UnreachableMember after remove-shard-replica.<br/> This is also similar to <a href="https://jira.opendaylight.org/browse/CONTROLLER-1705" title="Client on a follower prefix-based shard sometimes does not reconnect when the previous leader is isolated" class="issue-link" data-issue-key="CONTROLLER-1705"><del>CONTROLLER-1705</del></a> in that the client (at member-3) has not properly reconnected to the new leader.</p> <p>The quoted part of the title is seen in member-3 karaf.log <span class="error">&#91;1&#93;</span>:<br/> 2017-06-08 07:26:38,108 | INFO | lt-dispatcher-29 | aftActorLeadershipTransferCohort | 193 - org.opendaylight.controller.sal-akka-raft - 1.5.1.Carbon | member-3-shard-default-config: Successfully transferred leadership to null in 3.074 s</p> <p><span class="error">&#91;0&#93;</span> <a href="https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/736/log.html.gz#s1-s20-t1-k2-k9" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/736/log.html.gz#s1-s20-t1-k2-k9</a><br/> <span class="error">&#91;1&#93;</span> <a href="https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/736/odl3_karaf.log.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/736/odl3_karaf.log.gz</a></p> <p>Operating System: All<br/> Platform: All</p> CONTROLLER-1713

RequestTimeoutException after remove-shard-replica with "transferred leadership to null"

Bug Verified Done Robert Varga Vratko Polak Thu, 8 Jun 2017 15:38:45 +0000 Tue, 25 Jul 2023 08:24:42 +0000 Wed, 13 Sep 2017 08:30:41 +0000 clustering 0 5 <p>Something similar happened <span class="error">&#91;2&#93;</span> again. Karaf.log <span class="error">&#91;3&#93;</span> contains the quoted part, but there are other suspicious lines, so I am not really sure this is the same Bug.</p> <p>2017-06-11 11:53:06,783 | INFO | rd-dispatcher-38 | aftActorLeadershipTransferCohort | 193 - org.opendaylight.controller.sal-akka-raft - 1.5.1.Carbon | member-1-shard-default-config: Successfully transferred leadership to null in 2.319 s</p> <p><span class="error">&#91;2&#93;</span> <a href="https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/739/log.html.gz#s1-s36-t3-k2-k14" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/739/log.html.gz#s1-s36-t3-k2-k14</a><br/> <span class="error">&#91;3&#93;</span> <a href="https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/739/odl1_karaf.log.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/739/odl1_karaf.log.gz</a></p> <p>Examining history of <del>all</del> job, I have found more recent (2017-08-24) occurrence <span class="error">&#91;4&#93;</span>.</p> <p>After comparing karaf logs, I think this has to do with frontend reconnect happening while there is a member in PreLeader state.</p> <p>First, let me copy INFO logs (copied from <span class="error">&#91;5&#93;</span>) from a run where the test passed. The log belongs to member-1 (the old leader), the new leader is member-2. It is not clear from INFO level which member-1 component has first become aware that member-2 is a leader to allow successful reconnect (09:14:33,018), as member-1 Shard switched from Leader to Follower only after seeing (09:14:34,006) higher term from the other follower (member-3) AppendEntriesReply. This probably implies there was a time interval when both member-2 and member-1 claimed to be a Leader.</p> <p>2017-09-05 09:14:32,878 | INFO | qtp481975108-122 | ClusterAdminRpcService | 182 - org.opendaylight.controller.sal-cluster-admin-impl - 1.5.2.Carbon | Removing replica for shard default memberName member-1, datastoreType Config<br/> 2017-09-05 09:14:32,896 | INFO | ult-dispatcher-2 | ClusterAdminRpcService | 182 - org.opendaylight.controller.sal-cluster-admin-impl - 1.5.2.Carbon | Successfully removed replica for shard default<br/> 2017-09-05 09:14:32,906 | INFO | rd-dispatcher-40 | tActorServerConfigurationSupport | 174 - org.opendaylight.controller.sal-akka-raft - 1.5.2.Carbon | member-1-shard-default-config: ServerConfigurationPayload [serverConfig=[ServerInfo <span class="error">&#91;id=member-3-shard-default-config, isVoting=true&#93;</span>, ServerInfo <span class="error">&#91;id=member-2-shard-default-config, isVoting=true&#93;</span>]] has been successfully replicated to a majority of followers<br/> 2017-09-05 09:14:32,917 | INFO | rd-dispatcher-39 | ShardManager | 180 - org.opendaylight.controller.sal-distributed-datastore - 1.5.2.Carbon | shard-manager-config: Received LeaderStateChanged message: LeaderStateChanged <span class="error">&#91;memberId=member-1-shard-default-config, leaderId=null, leaderPayloadVersion=5&#93;</span><br/> 2017-09-05 09:14:32,943 | INFO | lt-dispatcher-21 | ClientActorBehavior | 178 - org.opendaylight.controller.cds-access-client - 1.1.2.Carbon | member-1-frontend-datastore-config: connection ConnectedClientConnection{client=ClientIdentifier</p> {frontend=member-1-frontend-datastore-config, generation=0}, cookie=0, backend=ShardBackendInfo{actor=Actor<a href="#-543839374" target="_blank" rel="noopener">akka://opendaylight-cluster-data/user/shardmanager-config/member-1-shard-default-config#-543839374</a>, sessionId=0, version=BORON, maxMessages=1000, cookie=0, shard=default, dataTree=present}} indicated no leadership, reconnecting it<br/> org.opendaylight.controller.cluster.access.commands.NotLeaderException: Actor Actor<a href="#-543839374" target="_blank" rel="noopener">akka://opendaylight-cluster-data/user/shardmanager-config/member-1-shard-default-config#-543839374</a> is not the current leader<br/> 2017-09-05 09:14:32,960 | INFO | lt-dispatcher-21 | ClientActorBehavior | 178 - org.opendaylight.controller.cds-access-client - 1.1.2.Carbon | member-1-frontend-datastore-config: connection ConnectedClientConnection{client=ClientIdentifier{frontend=member-1-frontend-datastore-config, generation=0} <p>, cookie=0, backend=ShardBackendInfo{actor=Actor<a href="#-543839374" target="_blank" rel="noopener">akka://opendaylight-cluster-data/user/shardmanager-config/member-1-shard-default-config#-543839374</a>, sessionId=0, version=BORON, maxMessages=1000, cookie=0, shard=default, dataTree=present}} reconnecting as ReconnectingClientConnection{client=ClientIdentifier</p> {frontend=member-1-frontend-datastore-config, generation=0}, cookie=0, backend=ShardBackendInfo{actor=Actor<a href="#-543839374" target="_blank" rel="noopener">akka://opendaylight-cluster-data/user/shardmanager-config/member-1-shard-default-config#-543839374</a>, sessionId=0, version=BORON, maxMessages=1000, cookie=0, shard=default, dataTree=present}}<br/> 2017-09-05 09:14:32,960 | INFO | lt-dispatcher-21 | ClientActorBehavior | 178 - org.opendaylight.controller.cds-access-client - 1.1.2.Carbon | member-1-frontend-datastore-config: refreshing backend for shard 0<br/> 2017-09-05 09:14:32,996 | INFO | lt-dispatcher-41 | ClientActorBehavior | 178 - org.opendaylight.controller.cds-access-client - 1.1.2.Carbon | member-1-frontend-datastore-config: resolved shard 0 to ShardBackendInfo{actor=Actor[akka.tcp://opendaylight-cluster-data@10.29.15.180:2550/user/shardmanager-config/member-2-shard-default-config#-754756015], sessionId=2, version=BORON, maxMessages=1000, cookie=0, shard=default, dataTree=absent}<br/> 2017-09-05 09:14:32,996 | INFO | lt-dispatcher-41 | ClientActorBehavior | 178 - org.opendaylight.controller.cds-access-client - 1.1.2.Carbon | member-1-frontend-datastore-config: resolving connection ReconnectingClientConnection{client=ClientIdentifier{frontend=member-1-frontend-datastore-config, generation=0} <p>, cookie=0, backend=ShardBackendInfo{actor=Actor<a href="#-543839374" target="_blank" rel="noopener">akka://opendaylight-cluster-data/user/shardmanager-config/member-1-shard-default-config#-543839374</a>, sessionId=0, version=BORON, maxMessages=1000, cookie=0, shard=default, dataTree=present}} to ConnectedClientConnection{client=ClientIdentifier</p> {frontend=member-1-frontend-datastore-config, generation=0}, cookie=0, backend=ShardBackendInfo{actor=Actor<span class="error">&#91;akka.tcp://opendaylight-cluster-data@10.29.15.180:2550/user/shardmanager-config/member-2-shard-default-config#-754756015&#93;</span>, sessionId=2, version=BORON, maxMessages=1000, cookie=0, shard=default, dataTree=absent}}<br/> 2017-09-05 09:14:33,018 | INFO | lt-dispatcher-41 | ClientActorBehavior | 178 - org.opendaylight.controller.cds-access-client - 1.1.2.Carbon | member-1-frontend-datastore-config: replaced connection ReconnectingClientConnection{client=ClientIdentifier{frontend=member-1-frontend-datastore-config, generation=0} <p>, cookie=0, backend=ShardBackendInfo{actor=Actor<a href="#-543839374" target="_blank" rel="noopener">akka://opendaylight-cluster-data/user/shardmanager-config/member-1-shard-default-config#-543839374</a>, sessionId=0, version=BORON, maxMessages=1000, cookie=0, shard=default, dataTree=present}} with ConnectedClientConnection{client=ClientIdentifier</p> {frontend=member-1-frontend-datastore-config, generation=0}, cookie=0, backend=ShardBackendInfo{actor=Actor<span class="error">&#91;akka.tcp://opendaylight-cluster-data@10.29.15.180:2550/user/shardmanager-config/member-2-shard-default-config#-754756015&#93;</span>, sessionId=2, version=BORON, maxMessages=1000, cookie=0, shard=default, dataTree=absent}} in 21.22 ms<br/> 2017-09-05 09:14:34,006 | INFO | rd-dispatcher-39 | Shard | 173 - org.opendaylight.controller.sal-clustering-commons - 1.5.2.Carbon | member-1-shard-default-config (Leader): Term 2 in "AppendEntriesReply <span class="error">&#91;term=2, success=false, followerId=member-3-shard-default-config, logLastIndex=12193, logLastTerm=2, forceInstallSnapshot=false, payloadVersion=5, raftVersion=3&#93;</span>" message is greater than leader's term 1 - switching to Follower<br/> 2017-09-05 09:14:34,007 | INFO | rd-dispatcher-39 | Shard | 173 - org.opendaylight.controller.sal-clustering-commons - 1.5.2.Carbon | member-1-shard-default-config (Leader) :- Switching from behavior Leader to Follower, election term: 2<br/> 2017-09-05 09:14:34,008 | INFO | lt-dispatcher-22 | RoleChangeNotifier | 173 - org.opendaylight.controller.sal-clustering-commons - 1.5.2.Carbon | RoleChangeNotifier for member-1-shard-default-config , received role change from Leader to Follower<br/> 2017-09-05 09:14:34,008 | INFO | rd-dispatcher-54 | ShardManager | 180 - org.opendaylight.controller.sal-distributed-datastore - 1.5.2.Carbon | shard-manager-config: Received LeaderStateChanged message: LeaderStateChanged <span class="error">&#91;memberId=member-1-shard-default-config, leaderId=null, leaderPayloadVersion=-1&#93;</span><br/> 2017-09-05 09:14:34,008 | INFO | rd-dispatcher-54 | ShardManager | 180 - org.opendaylight.controller.sal-distributed-datastore - 1.5.2.Carbon | shard-manager-config: Received role changed for member-1-shard-default-config from Leader to Follower<br/> 2017-09-05 09:14:34,969 | INFO | rd-dispatcher-37 | aftActorLeadershipTransferCohort | 174 - org.opendaylight.controller.sal-akka-raft - 1.5.2.Carbon | member-1-shard-default-config: Successfully transferred leadership to null in 2.052 s<br/> <br/> <br/> Now, when this Bug happens, the old leader (member-1) INFO messages (copied from karaf log <span class="error">&#91;6&#93;</span>) are nearly the same, except the unknown component never learns who the new leader (member-3) is, frontend keeps trying to connect to member-1 and never succeeds.<br/> <br/> 2017-08-24 15:24:13,620 | INFO | qtp412422320-101 | ClusterAdminRpcService | 182 - org.opendaylight.controller.sal-cluster-admin-impl - 1.5.2.Carbon | Removing replica for shard default memberName member-1, datastoreType Config<br/> 2017-08-24 15:24:13,654 | INFO | lt-dispatcher-26 | ClusterAdminRpcService | 182 - org.opendaylight.controller.sal-cluster-admin-impl - 1.5.2.Carbon | Successfully removed replica for shard default<br/> 2017-08-24 15:24:13,670 | INFO | rd-dispatcher-53 | tActorServerConfigurationSupport | 174 - org.opendaylight.controller.sal-akka-raft - 1.5.2.Carbon | member-1-shard-default-config: ServerConfigurationPayload [serverConfig=[ServerInfo <span class="error">&#91;id=member-3-shard-default-config, isVoting=true&#93;</span>, ServerInfo <span class="error">&#91;id=member-2-shard-default-config, isVoting=true&#93;</span>]] has been successfully replicated to a majority of followers<br/> 2017-08-24 15:24:13,681 | INFO | rd-dispatcher-37 | ShardManager | 180 - org.opendaylight.controller.sal-distributed-datastore - 1.5.2.Carbon | shard-manager-config: Received LeaderStateChanged message: LeaderStateChanged <span class="error">&#91;memberId=member-1-shard-default-config, leaderId=null, leaderPayloadVersion=5&#93;</span><br/> 2017-08-24 15:24:13,684 | INFO | lt-dispatcher-25 | ClientActorBehavior | 178 - org.opendaylight.controller.cds-access-client - 1.1.2.Carbon | member-1-frontend-datastore-config: connection ConnectedClientConnection{client=ClientIdentifier{frontend=member-1-frontend-datastore-config, generation=0} <p>, cookie=0, backend=ShardBackendInfo{actor=Actor<a href="#513397858" target="_blank" rel="noopener">akka://opendaylight-cluster-data/user/shardmanager-config/member-1-shard-default-config#513397858</a>, sessionId=0, version=BORON, maxMessages=1000, cookie=0, shard=default, dataTree=present}} indicated no leadership, reconnecting it<br/> org.opendaylight.controller.cluster.access.commands.NotLeaderException: Actor Actor<a href="#513397858" target="_blank" rel="noopener">akka://opendaylight-cluster-data/user/shardmanager-config/member-1-shard-default-config#513397858</a> is not the current leader<br/> 2017-08-24 15:24:13,691 | INFO | lt-dispatcher-25 | ClientActorBehavior | 178 - org.opendaylight.controller.cds-access-client - 1.1.2.Carbon | member-1-frontend-datastore-config: connection ConnectedClientConnection{client=ClientIdentifier</p> {frontend=member-1-frontend-datastore-config, generation=0}, cookie=0, backend=ShardBackendInfo{actor=Actor<a href="#513397858" target="_blank" rel="noopener">akka://opendaylight-cluster-data/user/shardmanager-config/member-1-shard-default-config#513397858</a>, sessionId=0, version=BORON, maxMessages=1000, cookie=0, shard=default, dataTree=present}} reconnecting as ReconnectingClientConnection{client=ClientIdentifier{frontend=member-1-frontend-datastore-config, generation=0} <p>, cookie=0, backend=ShardBackendInfo{actor=Actor<a href="#513397858" target="_blank" rel="noopener">akka://opendaylight-cluster-data/user/shardmanager-config/member-1-shard-default-config#513397858</a>, sessionId=0, version=BORON, maxMessages=1000, cookie=0, shard=default, dataTree=present}}<br/> 2017-08-24 15:24:13,691 | INFO | lt-dispatcher-25 | ClientActorBehavior | 178 - org.opendaylight.controller.cds-access-client - 1.1.2.Carbon | member-1-frontend-datastore-config: refreshing backend for shard 0<br/> 2017-08-24 15:24:13,744 | INFO | rd-dispatcher-53 | Shard | 173 - org.opendaylight.controller.sal-clustering-commons - 1.5.2.Carbon | member-1-shard-default-config: not currently leader, rejecting request ConnectClientRequest{target=ClientIdentifier</p> {frontend=member-1-frontend-datastore-config, generation=0} <p>, sequence=0, replyTo=Actor<span class="error">&#91;akka://opendaylight-cluster-data/temp/$q&#93;</span>, minVersion=BORON, maxVersion=BORON}. isLeader: true, isLeaderActive: false,isLeadershipTransferInProgress: true.<br/> 2017-08-24 15:24:13,748 | WARN | monPool-worker-3 | AbstractShardBackendResolver | 180 - org.opendaylight.controller.sal-distributed-datastore - 1.5.2.Carbon | Failed to resolve shard<br/> java.util.concurrent.TimeoutException: Connection attempt failed</p> <p>Here is karaf log <span class="error">&#91;7&#93;</span> segment from the new leader (member-3) showing transient PreLeader state.</p> <p>2017-08-24 15:24:13,680 | INFO | rd-dispatcher-31 | ShardManager | 180 - org.opendaylight.controller.sal-distributed-datastore - 1.5.2.Carbon | shard-manager-config: Received LeaderStateChanged message: LeaderStateChanged <span class="error">&#91;memberId=member-3-shard-default-config, leaderId=null, leaderPayloadVersion=5&#93;</span><br/> 2017-08-24 15:24:13,700 | INFO | rd-dispatcher-29 | Shard | 173 - org.opendaylight.controller.sal-clustering-commons - 1.5.2.Carbon | member-3-shard-default-config (Candidate): Starting new election term 2<br/> 2017-08-24 15:24:13,700 | INFO | rd-dispatcher-29 | Shard | 173 - org.opendaylight.controller.sal-clustering-commons - 1.5.2.Carbon | member-3-shard-default-config (Follower) :- Switching from behavior Follower to Candidate, election term: 2<br/> 2017-08-24 15:24:13,701 | INFO | lt-dispatcher-21 | RoleChangeNotifier | 173 - org.opendaylight.controller.sal-clustering-commons - 1.5.2.Carbon | RoleChangeNotifier for member-3-shard-default-config , received role change from Follower to Candidate<br/> 2017-08-24 15:24:13,701 | INFO | rd-dispatcher-33 | ShardManager | 180 - org.opendaylight.controller.sal-distributed-datastore - 1.5.2.Carbon | shard-manager-config: Received LeaderStateChanged message: LeaderStateChanged <span class="error">&#91;memberId=member-3-shard-default-config, leaderId=null, leaderPayloadVersion=-1&#93;</span><br/> 2017-08-24 15:24:13,701 | INFO | rd-dispatcher-33 | ShardManager | 180 - org.opendaylight.controller.sal-distributed-datastore - 1.5.2.Carbon | shard-manager-config: Received role changed for member-3-shard-default-config from Follower to Candidate<br/> 2017-08-24 15:24:13,789 | INFO | rd-dispatcher-29 | Shard | 173 - org.opendaylight.controller.sal-clustering-commons - 1.5.2.Carbon | member-3-shard-default-config (Candidate): LastApplied index 10366 is behind last index 10376 - switching to PreLeader<br/> 2017-08-24 15:24:13,795 | INFO | rd-dispatcher-29 | Shard | 173 - org.opendaylight.controller.sal-clustering-commons - 1.5.2.Carbon | member-3-shard-default-config (Candidate) :- Switching from behavior Candidate to PreLeader, election term: 2<br/> 2017-08-24 15:24:13,813 | INFO | rd-dispatcher-32 | ShardManager | 180 - org.opendaylight.controller.sal-distributed-datastore - 1.5.2.Carbon | shard-manager-config: Received LeaderStateChanged message: LeaderStateChanged <span class="error">&#91;memberId=member-3-shard-default-config, leaderId=member-3-shard-default-config, leaderPayloadVersion=5&#93;</span><br/> 2017-08-24 15:24:13,816 | INFO | lt-dispatcher-20 | RoleChangeNotifier | 173 - org.opendaylight.controller.sal-clustering-commons - 1.5.2.Carbon | RoleChangeNotifier for member-3-shard-default-config , received role change from Candidate to PreLeader<br/> 2017-08-24 15:24:13,817 | INFO | rd-dispatcher-32 | ShardManager | 180 - org.opendaylight.controller.sal-distributed-datastore - 1.5.2.Carbon | shard-manager-config: Received role changed for member-3-shard-default-config from Candidate to PreLeader<br/> 2017-08-24 15:24:13,838 | INFO | rd-dispatcher-31 | Shard | 173 - org.opendaylight.controller.sal-clustering-commons - 1.5.2.Carbon | member-3-shard-default-config (PreLeader) :- Switching from behavior PreLeader to Leader, election term: 2<br/> 2017-08-24 15:24:13,839 | INFO | lt-dispatcher-43 | RoleChangeNotifier | 173 - org.opendaylight.controller.sal-clustering-commons - 1.5.2.Carbon | RoleChangeNotifier for member-3-shard-default-config , received role change from PreLeader to Leader<br/> 2017-08-24 15:24:13,839 | INFO | rd-dispatcher-32 | ShardManager | 180 - org.opendaylight.controller.sal-distributed-datastore - 1.5.2.Carbon | shard-manager-config: Received role changed for member-3-shard-default-config from PreLeader to Leader<br/> 2017-08-24 15:24:13,840 | INFO | rd-dispatcher-32 | ShardManager | 180 - org.opendaylight.controller.sal-distributed-datastore - 1.5.2.Carbon | shard-manager-config: All Shards are ready - data store config is ready, available count is 0<br/> 2017-08-24 15:24:14,714 | INFO | rd-dispatcher-32 | Shard | 173 - org.opendaylight.controller.sal-clustering-commons - 1.5.2.Carbon | member-3-shard-default-config (Leader): Cannot append entries because sender's term 1 is less than 2</p> <p><span class="error">&#91;4&#93;</span> <a href="https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-all-carbon/418/log.html.gz#s1-s20-t1-k2-k10-k2-k1" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-all-carbon/418/log.html.gz#s1-s20-t1-k2-k10-k2-k1</a><br/> <span class="error">&#91;5&#93;</span> <a href="https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-all-carbon/432/odl1_karaf.log.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-all-carbon/432/odl1_karaf.log.gz</a><br/> <span class="error">&#91;6&#93;</span> <a href="https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-all-carbon/418/odl1_karaf.log.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-all-carbon/418/odl1_karaf.log.gz</a><br/> <span class="error">&#91;7&#93;</span> <a href="https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-all-carbon/418/odl3_karaf.log.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-all-carbon/418/odl3_karaf.log.gz</a></p> <p>This looks like we are failing to invalidate the cache after we have shut down the local member.</p> <p>Relevant logs:</p> <p>2017-08-24 15:24:08,268 | INFO | h for user karaf | command | 498 - org.apache.karaf.log.command - 3.0.8 | ROBOT MESSAGE: Starting test Local_Leader_Shutdown<br/> 2017-08-24 15:24:13,620 | INFO | qtp412422320-101 | ClusterAdminRpcService | 182 - org.opendaylight.controller.sal-cluster-admin-impl - 1.5.2.Carbon | Removing replica for shard default memberName member-1, datastoreType Config<br/> 2017-08-24 15:24:13,654 | INFO | lt-dispatcher-26 | ClusterAdminRpcService | 182 - org.opendaylight.controller.sal-cluster-admin-impl - 1.5.2.Carbon | Successfully removed replica for shard default<br/> 2017-08-24 15:24:13,670 | INFO | rd-dispatcher-53 | tActorServerConfigurationSupport | 174 - org.opendaylight.controller.sal-akka-raft - 1.5.2.Carbon | member-1-shard-default-config: ServerConfigurationPayload [serverConfig=[ServerInfo <span class="error">&#91;id=member-3-shard-default-config, isVoting=true&#93;</span>, ServerInfo <span class="error">&#91;id=member-2-shard-default-config, isVoting=true&#93;</span>]] has been successfully replicated to a majority of followers<br/> 2017-08-24 15:24:13,681 | INFO | rd-dispatcher-37 | ShardManager | 180 - org.opendaylight.controller.sal-distributed-datastore - 1.5.2.Carbon | shard-manager-config: Received LeaderStateChanged message: LeaderStateChanged <span class="error">&#91;memberId=member-1-shard-default-config, leaderId=null, leaderPayloadVersion=5&#93;</span><br/> 2017-08-24 15:24:13,684 | INFO | lt-dispatcher-25 | ClientActorBehavior | 178 - org.opendaylight.controller.cds-access-client - 1.1.2.Carbon | member-1-frontend-datastore-config: connection ConnectedClientConnection{client=ClientIdentifier</p> {frontend=member-1-frontend-datastore-config, generation=0} <p>, cookie=0, backend=ShardBackendInfo{actor=Actor<a href="#513397858" target="_blank" rel="noopener">akka://opendaylight-cluster-data/user/shardmanager-config/member-1-shard-default-config#513397858</a>, sessionId=0, version=BORON, maxMessages=1000, cookie=0, shard=default, dataTree=present}} indicated no leadership, reconnecting it</p> <p>2017-08-24 15:24:14,715 | INFO | rd-dispatcher-35 | ShardManager | 180 - org.opendaylight.controller.sal-distributed-datastore - 1.5.2.Carbon | shard-manager-config: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-1-shard-default-config, leaderId=null, leaderPayloadVersi<br/> on=-1]<br/> 2017-08-24 15:24:14,715 | INFO | rd-dispatcher-35 | ShardManager | 180 - org.opendaylight.controller.sal-distributed-datastore - 1.5.2.Carbon | shard-manager-config: Received role changed for member-1-shard-default-config from Leader to Follower</p> <p>2017-08-24 15:24:15,708 | INFO | rd-dispatcher-36 | aftActorLeadershipTransferCohort | 174 - org.opendaylight.controller.sal-akka-raft - 1.5.2.Carbon | member-1-shard-default-config: Successfully transferred leadership to null in 2.034 s<br/> 2017-08-24 15:24:15,719 | INFO | rd-dispatcher-34 | Shard | 173 - org.opendaylight.controller.sal-clustering-commons - 1.5.2.Carbon | Stopping Shard member-1-shard-default-config</p> <p>After which we are continuing to get:</p> <p>2017-08-24 15:24:20,799 | WARN | monPool-worker-3 | AbstractShardBackendResolver | 180 - org.opendaylight.controller.sal-distributed-datastore - 1.5.2.Carbon | Failed to resolve shard<br/> java.util.concurrent.TimeoutException: Connection attempt failed<br/> at org.opendaylight.controller.cluster.databroker.actors.dds.AbstractShardBackendResolver.wrap(AbstractShardBackendResolver.java:129)<span class="error">&#91;180:org.opendaylight.controller.sal-distributed-datastore:1.5.2.Carbon&#93;</span><br/> at org.opendaylight.controller.cluster.databroker.actors.dds.AbstractShardBackendResolver.lambda$connectShard$2(AbstractShardBackendResolver.java:142)<span class="error">&#91;180:org.opendaylight.controller.sal-distributed-datastore:1.5.2.Carbon&#93;</span><br/> at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)<span class="error">&#91;:1.8.0_141&#93;</span><br/> at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736)<span class="error">&#91;:1.8.0_141&#93;</span><br/> at java.util.concurrent.CompletableFuture$Completion.exec(CompletableFuture.java:443)<span class="error">&#91;:1.8.0_141&#93;</span><br/> at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)<span class="error">&#91;:1.8.0_141&#93;</span><br/> at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)<span class="error">&#91;:1.8.0_141&#93;</span><br/> at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)<span class="error">&#91;:1.8.0_141&#93;</span><br/> at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)<span class="error">&#91;:1.8.0_141&#93;</span><br/> Caused by: akka.pattern.AskTimeoutException: Ask timed out on [ActorSelection<span class="error">&#91;Anchor(akka://opendaylight-cluster-data/), Path(/user/shardmanager-config/member-1-shard-default-config)&#93;</span>] after <span class="error">&#91;5000 ms&#93;</span>. Sender<span class="error">&#91;null&#93;</span> sent message of type "org.opendaylight.controller.cluster.access.commands.ConnectClientRequest".<br/> at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:604)<span class="error">&#91;154:com.typesafe.akka.actor:2.4.18&#93;</span><br/> at akka.actor.Scheduler$$anon$4.run(Scheduler.scala:126)<span class="error">&#91;154:com.typesafe.akka.actor:2.4.18&#93;</span><br/> at scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:601)<span class="error">&#91;150:org.scala-lang.scala-library:2.11.11.v20170413-090219-8a413ba7cc&#93;</span><br/> at scala.concurrent.BatchingExecutor$class.execute(BatchingExecutor.scala:109)<span class="error">&#91;150:org.scala-lang.scala-library:2.11.11.v20170413-090219-8a413ba7cc&#93;</span><br/> at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:599)<span class="error">&#91;150:org.scala-lang.scala-library:2.11.11.v20170413-090219-8a413ba7cc&#93;</span><br/> at akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:329)<span class="error">&#91;154:com.typesafe.akka.actor:2.4.18&#93;</span><br/> at akka.actor.LightArrayRevolverScheduler$$anon$4.executeBucket$1(LightArrayRevolverScheduler.scala:280)<span class="error">&#91;154:com.typesafe.akka.actor:2.4.18&#93;</span><br/> at akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:284)<span class="error">&#91;154:com.typesafe.akka.actor:2.4.18&#93;</span><br/> at akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:236)<span class="error">&#91;154:com.typesafe.akka.actor:2.4.18&#93;</span><br/> at java.lang.Thread.run(Thread.java:748)<span class="error">&#91;:1.8.0_141&#93;</span></p> <p>Which seems to be pointing to the local shard actor, which obviously, is not responding. I do not believe this is a problem in AbstractShardBackendResolver, but rather in ShardManager.</p> <p>To clarify the error resolution process (via actorContext.findPrimaryShardAsync(String)) is returning the local actor ref, eventhough it was already shut down.</p> <p>This means that either primaryShardInfoCache is not invalidated, or ShardManager is returning the local actor when queried via FindPrimary.</p> <p>In both cases this looks like a problem with ShardManager.</p> <p>Guesstimate fix: <a href="https://git.opendaylight.org/gerrit/62973" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/62973</a></p> <p>&gt; Guesstimate fix: <a href="https://git.opendaylight.org/gerrit/62973" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/62973</a></p> <p>Merged. Ongoing Sandbox test <span class="error">&#91;9&#93;</span> suggests that fixes this.</p> <p><span class="error">&#91;9&#93;</span> <a href="https://jenkins.opendaylight.org/sandbox/job/controller-csit-3node-ddb-clean-lead-shutdown-longevity-only-carbon/3/console" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/sandbox/job/controller-csit-3node-ddb-clean-lead-shutdown-longevity-only-carbon/3/console</a></p> <p>&gt; Sandbox test <span class="error">&#91;9&#93;</span> suggests that fixes this.</p> <p>Run #2 has failed without this fix, after 35 iterations which took around 75 minutes.<br/> Run #3 has passed with this fix, after 23 hours. Setting to VERIFIED.</p> Development External issue ID 8639 External issue URL Issue Type ODL SR Target Milestone Rank 0|i02s9r: