https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/CONTROLLER-1670 controller <p>This manifests as a failure controller-csit-3node-drb-partnheal-longevity-only-carbon/8.</p> <p>It is a longevity test, the failure happened in iteration 4. Member-1 and member-2 had registered RPC implementations (returning constant-1 and constant-2 respectively). Member-3 did not have any implementation registered, and at the start of iteration, it was returning constant-2.<br/> The test then isolates member-2 and waits for member-3 to start returning constant-1.</p> <p>It is expected that after member-2 rejoins, member-3 should keep responding with constant-1, but the test encountered <span class="error">&#91;0&#93;</span> 501.</p> <p>I suspect this recent fix <span class="error">&#91;1&#93;</span> is the cause, and should be fixed.</p> <p><span class="error">&#91;0&#93;</span> <a href="https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-drb-partnheal-longevity-only-carbon/8/archives/log.html.gz#s1-t1-k3-k1-k1-k1-k1-k1-k1-k2-k1-k1-k6-k1-k1-k1-k1-k1-k1-k1-k1-k1-k1-k3-k1-k1-k1-k2-k1-k4-k7-k1" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-drb-partnheal-longevity-only-carbon/8/archives/log.html.gz#s1-t1-k3-k1-k1-k1-k1-k1-k1-k2-k1-k1-k6-k1-k1-k1-k1-k1-k1-k1-k1-k1-k1-k3-k1-k1-k1-k2-k1-k4-k7-k1</a><br/> <span class="error">&#91;1&#93;</span> <a href="https://git.opendaylight.org/gerrit/56783" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/56783</a></p> <p>Operating System: All<br/> Platform: All</p> CONTROLLER-1670

Rejoining member can make other member forget remotely registered RPC

Bug Resolved Won't Do Unassigned Vratko Polak Fri, 12 May 2017 09:58:23 +0000 Tue, 25 Jul 2023 08:24:36 +0000 Thu, 24 Aug 2017 13:10:05 +0000 clustering 0 3 <p>Seems like member-1 was briefly unreachable on member-3 which would explain the forgotten rpc registration</p> <p>This is still happening occasionally.<br/> Robot symptom: <span class="error">&#91;2&#93;</span>, this time in functional test.<br/> Corresponding karaf.log <span class="error">&#91;3&#93;</span> segment:<br/> 2017-05-22 19:01:59,364 | INFO | h for user karaf | command | 265 - org.apache.karaf.log.command - 3.0.8 | ROBOT MESSAGE: Starting test Invoke_Rpc_On_Each_Node_Again<br/> 2017-05-22 19:01:59,447 | INFO | lt-dispatcher-32 | ShardManager | 199 - org.opendaylight.controller.sal-distributed-datastore - 1.5.0.Carbon | Received UnreachableMember: memberName MemberName</p> {name=member-1} <p>, address: akka.tcp://opendaylight-cluster-data@10.29.12.72:2550</p> <p>Note that there were messages like below. It is not clear whether 501 can happen even without UnreachableMember if Robot hits the time after shard stabilization, but before first reachable gossip.<br/> 2017-05-22 19:01:58,442 | INFO | lt-dispatcher-22 | kka://opendaylight-cluster-data) | 174 - com.typesafe.akka.slf4j - 2.4.17 | Cluster Node <span class="error">&#91;akka.tcp://opendaylight-cluster-data@10.29.13.34:2550&#93;</span> - Ignoring received gossip from unreachable <span class="error">&#91;UniqueAddress(akka.tcp://opendaylight-cluster-data@10.29.12.94:2550,1537119417)&#93;</span> </p> <p><span class="error">&#91;2&#93;</span> <a href="https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/719/archives/log.html.gz#s1-s4-t6-k2-k1-k1-k1-k1-k1-k1-k1-k1-k1-k1-k3-k1-k1-k1-k2-k1-k4-k7-k1" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/719/archives/log.html.gz#s1-s4-t6-k2-k1-k1-k1-k1-k1-k1-k1-k1-k1-k1-k3-k1-k1-k1-k2-k1-k4-k7-k1</a><br/> <span class="error">&#91;3&#93;</span> <a href="https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/719/archives/odl3_karaf.log.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/719/archives/odl3_karaf.log.gz</a></p> <p>Happened again on Sandbox. UnreachableMember happened briefly, just after rejoining the member. Karaf.log segment from member-3, a surviving member reacting to member-2 rejoining:<br/> 2017-05-25 21:04:49,363 | INFO | ult-dispatcher-4 | ShardManager | 199 - org.opendaylight.controller.sal-distributed-datastore - 1.5.0.SNAPSHOT | shard-manager-operational: Received role changed for member-3-shard-toaster-operational from Leader to Follower<br/> 2017-05-25 21:04:49,739 | INFO | ult-dispatcher-4 | kka://opendaylight-cluster-data) | 174 - com.typesafe.akka.slf4j - 2.4.17 | Cluster Node <span class="error">&#91;akka.tcp://opendaylight-cluster-data@10.29.14.193:2550&#93;</span> - Marking node(s) as REACHABLE <span class="error">&#91;Member(address = akka.tcp://opendaylight-cluster-data@10.29.13.208:2550, status = Up)&#93;</span>. Node roles <span class="error">&#91;member-3&#93;</span><br/> 2017-05-25 21:04:51,563 | INFO | lt-dispatcher-26 | ShardManager | 199 - org.opendaylight.controller.sal-distributed-datastore - 1.5.0.SNAPSHOT | Received UnreachableMember: memberName MemberName</p> {name=member-2}, address: akka.tcp://opendaylight-cluster-data@10.29.13.49:2550<br/> 2017-05-25 21:04:51,563 | INFO | ult-dispatcher-6 | ShardManager | 199 - org.opendaylight.controller.sal-distributed-datastore - 1.5.0.SNAPSHOT | Received UnreachableMember: memberName MemberName{name=member-2} <p>, address: akka.tcp://opendaylight-cluster-data@10.29.13.49:2550<br/> 2017-05-25 21:04:51,563 | INFO | lt-dispatcher-26 | EntityOwnershipShard | 192 - org.opendaylight.controller.sal-clustering-commons - 1.5.0.SNAPSHOT | member-3-shard-entity-ownership-operational: onPeerDown: PeerDown <span class="error">&#91;memberName=member-2, peerId=member-2-shard-entity-ownership-operational&#93;</span><br/> 2017-05-25 21:04:52,549 | INFO | ult-dispatcher-2 | ShardManager | 199 - org.opendaylight.controller.sal-distributed-datastore - 1.5.0.SNAPSHOT | Received ReachableMember: memberName MemberName</p> {name=member-2}, address: akka.tcp://opendaylight-cluster-data@10.29.13.49:2550<br/> 2017-05-25 21:04:52,549 | INFO | ult-dispatcher-2 | ShardInformation | 199 - org.opendaylight.controller.sal-distributed-datastore - 1.5.0.SNAPSHOT | updatePeerAddress for peer member-2-shard-default-config with address akka.tcp://opendaylight-cluster-data@10.29.13.49:2550/user/shardmanager-config/member-2-shard-default-config<br/> 2017-05-25 21:04:52,549 | INFO | lt-dispatcher-29 | ShardManager | 199 - org.opendaylight.controller.sal-distributed-datastore - 1.5.0.SNAPSHOT | Received ReachableMember: memberName MemberName{name=member-2} <p>, address: akka.tcp://opendaylight-cluster-data@10.29.13.49:2550</p> <p>Member-2 was isolated for around 41 seconds.</p> <p>Happened <span class="error">&#91;4&#93;</span> on current stable/carbon.</p> <p><span class="error">&#91;4&#93;</span> <a href="https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-drb-partnheal-longevity-only-carbon/12/archives/log.html.gz#s1-t1-k3-k1-k1-k1-k1-k1-k1-k2-k1-k1-k6-k1-k1-k1-k1-k1-k1-k2-k1-k1-k1-k3-k1-k1-k1-k2-k1-k4-k7-k1" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-drb-partnheal-longevity-only-carbon/12/archives/log.html.gz#s1-t1-k3-k1-k1-k1-k1-k1-k1-k2-k1-k1-k6-k1-k1-k1-k1-k1-k1-k2-k1-k1-k1-k3-k1-k1-k1-k2-k1-k4-k7-k1</a></p> <p>Also this week: <a href="https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-drb-partnheal-longevity-only-carbon/13/console.log.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-drb-partnheal-longevity-only-carbon/13/console.log.gz</a></p> <p>&gt; UnreachableMember happened briefly, just after rejoining the member.</p> <p>Using a logging change <span class="error">&#91;6&#93;</span> I have identified few lines in huge log <span class="error">&#91;7&#93;</span> related to the robot failure <span class="error">&#91;8&#93;</span>. This is around two seconds after the test rejoined member-2 (10.29.15.65), when member-3 (10.29.15.201 without local RPC registation) obtains gossip from the rejoining member-2, who has not yet detected member-1 (10.29.15.65) is reachable. It seems member-3 considers member-2 to be more up-to-date and marks member-1 as unreachable, thus losing access to remote RPC registration. Although it is possible member-3 already had wrong local reachability information for some reason.</p> <p>From akka code side, the long "merge" line is reported here <span class="error">&#91;9&#93;</span> referring to the computation done here <span class="error">&#91;10&#93;</span>.<br/> The identified lines:</p> <p>2017-07-25 12:05:27,768 | DEBUG | lt-dispatcher-39 | EndpointWriter | 174 - com.typesafe.akka.slf4j - 2.4.18 | received local message RemoteMessage: [ActorSelectionMessage(akka.cluster.GossipEnvelope@384a4579,Vector(system, cluster, cor<br/> e, daemon),false)] to [Actor<span class="error">&#91;akka://opendaylight-cluster-data/&#93;</span>]&lt;+<span class="error">&#91;akka://opendaylight-cluster-data/&#93;</span> from [Actor<span class="error">&#91;akka.tcp://opendaylight-cluster-data@10.29.13.236:2550/system/cluster/core/daemon#975906890&#93;</span>()]<br/> 2017-07-25 12:05:27,775 | DEBUG | lt-dispatcher-39 | ClusterCoreDaemon | 174 - com.typesafe.akka.slf4j - 2.4.18 | Cluster Node <span class="error">&#91;akka.tcp://opendaylight-cluster-data@10.29.15.201:2550&#93;</span> - Receiving gossip from [UniqueAddress(akka.tcp://opend<br/> aylight-cluster-data@10.29.13.236:2550,1060309017)]<br/> 2017-07-25 12:05:27,778 | INFO | rd-dispatcher-41 | ShardManager | 199 - org.opendaylight.controller.sal-distributed-datastore - 1.5.2.SNAPSHOT | Received UnreachableMember: memberName MemberName</p> {name=member-1}, address: akka.tcp://o<br/> pendaylight-cluster-data@10.29.15.65:2550<br/> 2017-07-25 12:05:27,778 | INFO | rd-dispatcher-42 | ShardManager | 199 - org.opendaylight.controller.sal-distributed-datastore - 1.5.2.SNAPSHOT | Received UnreachableMember: memberName MemberName{name=member-1} <p>, address: akka.tcp://o<br/> pendaylight-cluster-data@10.29.15.65:2550<br/> 2017-07-25 12:05:27,778 | DEBUG | lt-dispatcher-20 | ClusterCoreDaemon | 174 - com.typesafe.akka.slf4j - 2.4.18 | Couldn't establish a causal relationship between "remote" gossip and "local" gossip - Remote[Gossip(members = [Member(address<br/> = akka.tcp://opendaylight-cluster-data@10.29.13.236:2550, status = Up), Member(address = akka.tcp://opendaylight-cluster-data@10.29.15.201:2550, status = Up), Member(address = akka.tcp://opendaylight-cluster-data@10.29.15.65:2550, status = Up)], overvie<br/> w = GossipOverview(reachability = [akka.tcp://opendaylight-cluster-data@10.29.13.236:2550 -&gt; akka.tcp://opendaylight-cluster-data@10.29.15.201:2550: Reachable <span class="error">&#91;Reachable&#93;</span> (5), akka.tcp://opendaylight-cluster-data@10.29.13.236:2550 -&gt; akka.tcp://opendayli<br/> ght-cluster-data@10.29.15.65:2550: Unreachable <span class="error">&#91;Unreachable&#93;</span> (3)], seen = <span class="error">&#91;UniqueAddress(akka.tcp://opendaylight-cluster-data@10.29.13.236:2550,1060309017)&#93;</span>), version = VectorClock(003f54fb4960bf1c7f7b77ad60d9c1eb -&gt; 5, 08e95a0c1ccb68238e08b7e282dd04df -<br/> &gt; 3, e2ecf54bf1a5bb7f8e94008a98ac3366 -&gt; 7))] - Local[Gossip(members = [Member(address = akka.tcp://opendaylight-cluster-data@10.29.13.236:2550, status = Up), Member(address = akka.tcp://opendaylight-cluster-data@10.29.15.201:2550, status = Up), Member(a<br/> ddress = akka.tcp://opendaylight-cluster-data@10.29.15.65:2550, status = Up)], overview = GossipOverview(reachability = [akka.tcp://opendaylight-cluster-data@10.29.15.65:2550 -&gt; akka.tcp://opendaylight-cluster-data@10.29.13.236:2550: Unreachable [Unreach<br/> able] (5)], seen = <span class="error">&#91;UniqueAddress(akka.tcp://opendaylight-cluster-data@10.29.15.201:2550,-3420732)&#93;</span>), version = VectorClock(003f54fb4960bf1c7f7b77ad60d9c1eb -&gt; 2, 08e95a0c1ccb68238e08b7e282dd04df -&gt; 5, e2ecf54bf1a5bb7f8e94008a98ac3366 -&gt; 8))] - merged th<br/> em into [Gossip(members = [Member(address = akka.tcp://opendaylight-cluster-data@10.29.13.236:2550, status = Up), Member(address = akka.tcp://opendaylight-cluster-data@10.29.15.201:2550, status = Up), Member(address = akka.tcp://opendaylight-cluster-data<br/> @10.29.15.65:2550, status = Up)], overview = GossipOverview(reachability = [akka.tcp://opendaylight-cluster-data@10.29.13.236:2550 -&gt; akka.tcp://opendaylight-cluster-data@10.29.15.201:2550: Reachable <span class="error">&#91;Reachable&#93;</span> (5), akka.tcp://opendaylight-cluster-data@<br/> 10.29.13.236:2550 -&gt; akka.tcp://opendaylight-cluster-data@10.29.15.65:2550: Unreachable <span class="error">&#91;Unreachable&#93;</span> (3), akka.tcp://opendaylight-cluster-data@10.29.15.65:2550 -&gt; akka.tcp://opendaylight-cluster-data@10.29.13.236:2550: Unreachable <span class="error">&#91;Unreachable&#93;</span> (5)], se<br/> en = []), version = VectorClock(003f54fb4960bf1c7f7b77ad60d9c1eb -&gt; 5, 08e95a0c1ccb68238e08b7e282dd04df -&gt; 5, e2ecf54bf1a5bb7f8e94008a98ac3366 -&gt; 8))]</p> <p><span class="error">&#91;6&#93;</span> <a href="https://git.opendaylight.org/gerrit/#/c/60727/1" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/60727/1</a><br/> <span class="error">&#91;7&#93;</span> <a href="https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-drb-partnheal-longevity-only-carbon/3/odl3_karaf.log.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-drb-partnheal-longevity-only-carbon/3/odl3_karaf.log.gz</a><br/> <span class="error">&#91;8&#93;</span> <a href="https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-drb-partnheal-longevity-only-carbon/3/log.html.gz#s1-t1-k3-k1-k1-k1-k1-k1-k1-k2-k1-k1-k6-k1-k1-k1-k1-k1-k1-k1-k1-k1-k1-k3-k1-k1-k1-k2-k1-k4-k7-k1" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-drb-partnheal-longevity-only-carbon/3/log.html.gz#s1-t1-k3-k1-k1-k1-k1-k1-k1-k2-k1-k1-k6-k1-k1-k1-k1-k1-k1-k1-k1-k1-k1-k3-k1-k1-k1-k2-k1-k4-k7-k1</a><br/> <span class="error">&#91;9&#93;</span> <a href="https://github.com/akka/akka/blob/v2.4.17/akka-cluster/src/main/scala/akka/cluster/ClusterDaemon.scala#L736-L742" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/akka/akka/blob/v2.4.17/akka-cluster/src/main/scala/akka/cluster/ClusterDaemon.scala#L736-L742</a><br/> <span class="error">&#91;10&#93;</span> <a href="https://github.com/akka/akka/blob/v2.4.17/akka-cluster/src/main/scala/akka/cluster/ClusterDaemon.scala#L704-L726" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/akka/akka/blob/v2.4.17/akka-cluster/src/main/scala/akka/cluster/ClusterDaemon.scala#L704-L726</a></p> <p>Intermediate churn is expected, as akka needs a few attempts to restore its vector clocks, during which time it may report incorrect reachability status.</p> Development External issue ID 8430 External issue URL Issue Type Rank 0|i02s07: