https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/CONTROLLER-2036 controller <p>Found in <a href="https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-3node-clustering-bulkomatic-only-sulfur/191/" class="external-link" target="_blank" rel="nofollow noopener">openflowplugin-csit-3node-clustering-bulkomatic-only-sulfur/191/</a></p> <p>After cluster is restarted, one instance generates the following warning message repeatedly for about 5 minutes until the leader is killed in a different test case.:</p> <p><a href="https://s3-logs.opendaylight.org/logs/releng/vex-yul-odl-jenkins-1/openflowplugin-csit-3node-clustering-bulkomatic-only-sulfur/191/odl_3/odl3_karaf.log.gz" class="external-link" target="_blank" rel="nofollow noopener">https://s3-logs.opendaylight.org/logs/releng/vex-yul-odl-jenkins-1/openflowplugin-csit-3node-clustering-bulkomatic-only-sulfur/191/odl_3/odl3_karaf.log.gz</a></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>2022-04-05T02:58:18,683 | WARN | opendaylight-cluster-data-akka.actor.default-dispatcher-45 | CandidateRegistryInit | 202 - org.opendaylight.controller.eos-dom-akka - 5.0.1 | member-3 : Initial removal of candidates from previous iteration failed. Rescheduling. java.util.concurrent.TimeoutException: Ask timed out on [Actor[akka://opendaylight-cluster-data/system/singletonProxyOwnerSupervisor-no-dc#2033451887]] after [5000 ms]. Message of type [org.opendaylight.controller.eos.akka.owner.supervisor.command.ClearCandidatesForMember]. A typical reason for `AskTimeoutException` is that the recipient actor didn't send a reply. at akka.actor.typed.scaladsl.AskPattern$.$anonfun$onTimeout$1(AskPattern.scala:131) ~[bundleFile:?] at akka.pattern.PromiseActorRef$.$anonfun$apply$1(AskSupport.scala:730) ~[bundleFile:?] at akka.actor.Scheduler$$anon$7.run(Scheduler.scala:479) ~[bundleFile:?] at scala.concurrent.ExecutionContext$parasitic$.execute(ExecutionContext.scala:222) ~[bundleFile:?] at akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:365) ~[bundleFile:?] at akka.actor.LightArrayRevolverScheduler$$anon$3.executeBucket$1(LightArrayRevolverScheduler.scala:314) ~[bundleFile:?] at akka.actor.LightArrayRevolverScheduler$$anon$3.nextTick(LightArrayRevolverScheduler.scala:318) ~[bundleFile:?] at akka.actor.LightArrayRevolverScheduler$$anon$3.run(LightArrayRevolverScheduler.scala:270) ~[bundleFile:?] at java.lang.Thread.run(Thread.java:829) ~[?:?] </pre> </div></div> CONTROLLER-2036

Failure of initial removal of candidates from previous iteration

Bug Medium Resolved Duplicate Sangwook Ha Sangwook Ha Tue, 5 Apr 2022 21:44:30 +0000 Wed, 6 Apr 2022 17:23:10 +0000 Wed, 6 Apr 2022 17:22:53 +0000 5.0.1 5.0.2 clustering 0 2 <p>Hmm, this looks rather weird. We are getting these:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>2022-04-05T03:03:15,442 | INFO | opendaylight-cluster-data-akka.actor.default-dispatcher-44 | LocalActorRef | 206 - org.opendaylight.controller.repackaged-akka - 5.0.1 | Message [akka.actor.ReceiveTimeout$] to Actor[akka://opendaylight-cluster-data/system/IO-TCP/selectors/$a/9#-1115372167] was not delivered. [295] dead letters encountered, of which 284 were not logged. The counter will be reset now. If this is not an expected behavior then Actor[akka://opendaylight-cluster-data/system/IO-TCP/selectors/$a/9#-1115372167] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. </pre> </div></div> <p>Hunting through int/test, it seems openflowplugin using is using an ancient akka.conf.template. Perhaps it needs an update?</p> <p>The template file is in the <tt>tools</tt> directory, outside of <tt>csit</tt>, and I think it's not really used for the test suite.<br/> Clustering configuration doesn't seem to be generated/modified once the configuration files are created during the initial controller installation for openflowplugin clustering tests.</p> <p>Interesting, I wonder where is that TCP reference coming from.<br/> Anyway, it would seem the problem is that the target actor is unreachable &#8211; and all we have is an ActorRef. We probably need to restart when the supervisor.<br/> I'll need to dig into a bit more.</p> <p>Actually <a href="https://s3-logs.opendaylight.org/logs/releng/vex-yul-odl-jenkins-1/openflowplugin-csit-3node-clustering-bulkomatic-only-sulfur/191/odl_1/odl1_karaf.log.gz" class="external-link" target="_blank" rel="nofollow noopener">https://s3-logs.opendaylight.org/logs/releng/vex-yul-odl-jenkins-1/openflowplugin-csit-3node-clustering-bulkomatic-only-sulfur/191/odl_1/odl1_karaf.log.gz</a> has:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>2022-04-05T02:58:10,693 | ERROR | opendaylight-cluster-data-akka.actor.default-dispatcher-32 | Behavior$ | 206 - org.opendaylight.controller.repackaged-akka - 5.0.1 | Supervisor StopSupervisor saw failure: Ask timed out on [Actor[akka://opendaylight-cluster-data/system/typedDdataReplicator#380268694]] after [5000 ms]. Message of type [akka.cluster.ddata.typed.javadsl.Replicator$Get]. A typical reason for `AskTimeoutException` is that the recipient actor didn't send a reply. java.util.concurrent.TimeoutException: Ask timed out on [Actor[akka://opendaylight-cluster-data/system/typedDdataReplicator#380268694]] after [5000 ms]. Message of type [akka.cluster.ddata.typed.javadsl.Replicator$Get]. A typical reason for `AskTimeoutException` is that the recipient actor didn't send a reply. at akka.actor.typed.scaladsl.AskPattern$.$anonfun$onTimeout$1(AskPattern.scala:131) ~[bundleFile:?] at akka.pattern.PromiseActorRef$.$anonfun$apply$1(AskSupport.scala:730) ~[bundleFile:?] at akka.actor.Scheduler$$anon$7.run(Scheduler.scala:479) ~[bundleFile:?] at scala.concurrent.ExecutionContext$parasitic$.execute(ExecutionContext.scala:222) ~[bundleFile:?] at akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:365) ~[bundleFile:?] at akka.actor.LightArrayRevolverScheduler$$anon$3.executeBucket$1(LightArrayRevolverScheduler.scala:314) ~[bundleFile:?] at akka.actor.LightArrayRevolverScheduler$$anon$3.nextTick(LightArrayRevolverScheduler.scala:318) ~[bundleFile:?] at akka.actor.LightArrayRevolverScheduler$$anon$3.run(LightArrayRevolverScheduler.scala:270) ~[bundleFile:?] at java.lang.Thread.run(Thread.java:829) ~[?:?] 2022-04-05T02:58:10,700 | INFO | opendaylight-cluster-data-akka.actor.default-dispatcher-14 | ClusterSingletonManager | 206 - org.opendaylight.controller.repackaged-akka - 5.0.1 | Singleton actor [akka://opendaylight-cluster-data/system/singletonManagerOwnerSupervisor/OwnerSupervisor] was terminated </pre> </div></div> Duplicate CONTROLLER-2035 Development Rank 0|i041zz: