https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/CONTROLLER-2032 controller <p>Build Version: <b>Phosphorus SR1</b></p> <p>3 Node Cluster</p> <p>From the Logs - <b>member_1</b> is chosen as the Master for the Devices </p> <p>Scenario:</p> <p>--------------------</p> <p>Triggered 200 Mount requests at the same time to ODL.</p> <p>In the logs, could see the requests are received and the Mount process was started for few of the devices, but after sometime we start getting the AskTimeoutException continuously which causes the further device mounts.</p> <p>When checked the mount status only around 78 devices among 200 were Connected, and other devices mount got stuck due to above, it dint even span the Netconf actor for the mounting. </p> <p> </p> <p>2022-02-22T20:43:22,747 | ERROR | opendaylight-cluster-data-akka.actor.default-dispatcher-46 | SupervisorStrategy               | 213 - org.opendaylight.controller.repackaged-akka - 4.0.7 | Ask timed out on <a href="#-1603400317" target="_blank" rel="noopener">Actor[akka://opendaylight-cluster-data/system/typedDdataReplicator#-1603400317]</a>] after <span class="error">&#91;5000 ms&#93;</span>. Message of type <span class="error">&#91;akka.cluster.ddata.typed.javadsl.Replicator$Get&#93;</span>. A typical reason for `AskTimeoutException` is that the recipient actor didn't send a reply.2022-02-22T20:43:22,747 | ERROR | opendaylight-cluster-data-akka.actor.default-dispatcher-46 | SupervisorStrategy               | 213 - org.opendaylight.controller.repackaged-akka - 4.0.7 | Ask timed out on <a href="#-1603400317" target="_blank" rel="noopener">Actor[akka://opendaylight-cluster-data/system/typedDdataReplicator#-1603400317]</a>] after <span class="error">&#91;5000 ms&#93;</span>. Message of type <span class="error">&#91;akka.cluster.ddata.typed.javadsl.Replicator$Get&#93;</span>. A typical reason for `AskTimeoutException` is that the recipient actor didn't send a reply.java.util.concurrent.TimeoutException: Ask timed out on <a href="#-1603400317" target="_blank" rel="noopener">Actor[akka://opendaylight-cluster-data/system/typedDdataReplicator#-1603400317]</a>] after <span class="error">&#91;5000 ms&#93;</span>. Message of type <span class="error">&#91;akka.cluster.ddata.typed.javadsl.Replicator$Get&#93;</span>. A typical reason for `AskTimeoutException` is that the recipient actor didn't send a reply. at akka.actor.typed.scaladsl.AskPattern$.$anonfun$onTimeout$1(AskPattern.scala:131) ~<span class="error">&#91;bundleFile:?&#93;</span> at akka.pattern.PromiseActorRef$.$anonfun$apply$1(AskSupport.scala:730) ~<span class="error">&#91;bundleFile:?&#93;</span> at akka.actor.Scheduler$$anon$7.run(Scheduler.scala:479) ~<span class="error">&#91;bundleFile:?&#93;</span> at scala.concurrent.ExecutionContext$parasitic$.execute(ExecutionContext.scala:222) ~<span class="error">&#91;bundleFile:?&#93;</span> at akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:365) ~<span class="error">&#91;bundleFile:?&#93;</span> at akka.actor.LightArrayRevolverScheduler$$anon$3.executeBucket$1(LightArrayRevolverScheduler.scala:314) ~<span class="error">&#91;bundleFile:?&#93;</span> at akka.actor.LightArrayRevolverScheduler$$anon$3.nextTick(LightArrayRevolverScheduler.scala:318) ~<span class="error">&#91;bundleFile:?&#93;</span> at akka.actor.LightArrayRevolverScheduler$$anon$3.run(LightArrayRevolverScheduler.scala:270) ~<span class="error">&#91;bundleFile:?&#93;</span> at java.lang.Thread.run(Thread.java:829) ~<span class="error">&#91;?:?&#93;</span>2022-02-22T20:43:22,788 | INFO  | opendaylight-cluster-data-akka.actor.default-dispatcher-18 | RemoteActorRefProvider$RemoteDeadLetterActorRef | 213 - org.opendaylight.controller.repackaged-akka - 4.0.7 | Message <span class="error">&#91;akka.cluster.ddata.typed.javadsl.Replicator$GetSuccess&#93;</span> to Actor<span class="error">&#91;akka://opendaylight-cluster-data/deadLetters&#93;</span> was not delivered. <span class="error">&#91;5&#93;</span> dead letters encountered. If this is not an expected behavior then Actor<span class="error">&#91;akka://opendaylight-cluster-data/deadLetters&#93;</span> may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.2022-02-22T20:43:22,726 | ERROR | opendaylight-cluster-data-akka.actor.default-dispatcher-15 | Behavior$                        | 213 - org.opendaylight.controller.repackaged-akka - 4.0.7 | Supervisor StopSupervisor saw failure: Ask timed out on <a href="#-1603400317" target="_blank" rel="noopener">Actor[akka://opendaylight-cluster-data/system/typedDdataReplicator#-1603400317]</a>] after <span class="error">&#91;5000 ms&#93;</span>. Message of type <span class="error">&#91;akka.cluster.ddata.typed.javadsl.Replicator$Update&#93;</span>. A typical reason for `AskTimeoutException` is that the recipient actor didn't send a reply.java.util.concurrent.TimeoutException: Ask timed out on <a href="#-1603400317" target="_blank" rel="noopener">Actor[akka://opendaylight-cluster-data/system/typedDdataReplicator#-1603400317]</a>] after <span class="error">&#91;5000 ms&#93;</span>. Message of type <span class="error">&#91;akka.cluster.ddata.typed.javadsl.Replicator$Update&#93;</span>. A typical reason for `AskTimeoutException` is that the recipient actor didn't send a reply. at akka.actor.typed.scaladsl.AskPattern$.$anonfun$onTimeout$1(AskPattern.scala:131) ~<span class="error">&#91;bundleFile:?&#93;</span> at akka.pattern.PromiseActorRef$.$anonfun$apply$1(AskSupport.scala:730) ~<span class="error">&#91;bundleFile:?&#93;</span> at akka.actor.Scheduler$$anon$7.run(Scheduler.scala:479) ~<span class="error">&#91;bundleFile:?&#93;</span> at scala.concurrent.ExecutionContext$parasitic$.execute(ExecutionContext.scala:222) ~<span class="error">&#91;bundleFile:?&#93;</span> at akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:365) ~<span class="error">&#91;bundleFile:?&#93;</span> at akka.actor.LightArrayRevolverScheduler$$anon$3.executeBucket$1(LightArrayRevolverScheduler.scala:314) ~<span class="error">&#91;bundleFile:?&#93;</span> at akka.actor.LightArrayRevolverScheduler$$anon$3.nextTick(LightArrayRevolverScheduler.scala:318) ~<span class="error">&#91;bundleFile:?&#93;</span> at akka.actor.LightArrayRevolverScheduler$$anon$3.run(LightArrayRevolverScheduler.scala:270) ~<span class="error">&#91;bundleFile:?&#93;</span> at java.lang.Thread.run(Thread.java:829) ~<span class="error">&#91;?:?&#93;</span>2022-02-22T20:43:22,805 | INFO  | opendaylight-cluster-data-akka.actor.default-dispatcher-16 | ClusterSingletonManager          | 213 - org.opendaylight.controller.repackaged-akka - 4.0.7 | Singleton actor <span class="error">&#91;akka://opendaylight-cluster-data/system/singletonManagerOwnerSupervisor/OwnerSupervisor&#93;</span> was terminated2022-02-22T20:43:22,806 | INFO  | opendaylight-cluster-data-akka.actor.default-dispatcher-16 | LocalActorRef                    | 213 - org.opendaylight.controller.repackaged-akka - 4.0.7 | Message <span class="error">&#91;akka.cluster.ddata.typed.javadsl.Replicator$Changed&#93;</span> wrapped in <span class="error">&#91;akka.actor.typed.internal.AdaptMessage&#93;</span> to Actor<a href="#901657858" target="_blank" rel="noopener">akka://opendaylight-cluster-data/system/singletonManagerOwnerSupervisor/OwnerSupervisor#901657858</a> was not delivered. <span class="error">&#91;6&#93;</span> dead letters encountered. If this is not an expected behavior then Actor<a href="#901657858" target="_blank" rel="noopener">akka://opendaylight-cluster-data/system/singletonManagerOwnerSupervisor/OwnerSupervisor#901657858</a> may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.2022-02-22T20:43:22,807 | INFO  | opendaylight-cluster-data-akka.actor.default-dispatcher-16 | LocalActorRef                    | 213 - org.opendaylight.controller.repackaged-akka - 4.0.7 | Message <span class="error">&#91;akka.cluster.ddata.typed.javadsl.Replicator$Changed&#93;</span> wrapped in <span class="error">&#91;akka.actor.typed.internal.AdaptMessage&#93;</span> to Actor<a href="#901657858" target="_blank" rel="noopener">akka://opendaylight-cluster-data/system/singletonManagerOwnerSupervisor/OwnerSupervisor#901657858</a> was not delivered. <span class="error">&#91;7&#93;</span> dead letters encountered. If this is not an expected behavior then Actor<a href="#901657858" target="_blank" rel="noopener">akka://opendaylight-cluster-data/system/singletonManagerOwnerSupervisor/OwnerSupervisor#901657858</a> may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.</p> <p> </p> <p> </p> <p> </p> <p> </p> <p> </p> CONTROLLER-2032

Phosphorus SR1 - AskTimeoutException while trying to mount multiple Netconf devices at a time

Bug Highest In Progress Unresolved Shibu Vijayakumar Shibu Vijayakumar Tue, 22 Feb 2022 21:18:18 +0000 Wed, 6 Apr 2022 17:14:15 +0000 4.0.10 clustering 0 5 <p><b>Major Concern on this Issue is:</b> Once we hit this timeout issue, no further mount request is processed by ODL.</p> <p>Tried giving a new POST mount request for a new device other than the above 200 - device vmx-201, api was passed, but the actual mount process was not started by the ODL. </p> <p>Even after trying Deleting the 200 devices and then trying a new mount dint go through. But delete process was successful.</p> <p> </p> <p>Device 201 Single mount after the 200 parallel mount was performed at <br/> <b>2022-02-22T20:48:14,434</b> | INFO | opendaylight-cluster-data-notification-dispatcher-59 | DefaultSchemaResourceManager | 287 - org.opendaylight.netconf.sal-netconf-connector - 2.0.11 | Netconf connector for device RemoteDevice{vmx-201} will use schema cache directory junos instead of schema</p> <p>From the logs it could see, DOMEntity of the EntityTypeListenerActor as </p> <p>_nodeId=Uri{_value=vmx-201}}]]}}]], state=REMOTE_OWNERSHIP_LOST_NO_OWNER <span class="error">&#91;wasOwner=false, isOwner=false, hasOwner=false&#93;</span></p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=rovarga" class="user-hover" rel="rovarga">rovarga</a> Another observation, once we get this error in the ODL instance which has the Master role was the devices mount, if we down that instance - could see another election happens and one of the other ODL gets the Leader role and after that it resumes all the above mounts that were stuck and all devices moves to Connected.</p> <p>So the ATE is troublesome, as it seems to indicate something in Akka replication. Two things:</p> <ol> <li>try to monitor the VMs/JVMs for signs of issues, most notably GC pauses and similar</li> <li>try the test case with odl-netconf-topology, i.e. without cluster, to bring down the amount of moving pieces</li> </ol> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=rovarga" class="user-hover" rel="rovarga">rovarga</a> Sure, will check on the point 1.</p> <p>W.r.t point 2 , have executed the test case and it was SUCCESS with no ATE in single instance ODL, enabling odl-netconf-topology, also in the Clustered ODL setup of 3 node, with the feature: odl-netconf-topology installed instead of "odl-netconf-clustered-topology".</p> <p>But the concern with the 3 node clustered ODL in the above case is the Netconf device Mount Point is created only in one of the instance where the POST mount api hits, and in the other 2 its not, and due to that the below GET and PUT netconf apis were failing in those instances, saying "mount point doesn't exists". </p> <p><b><em>Any workaround or suggestions on this scenario to make it work as a Cluster with the</em></b> <br/> <b><em>odl-netconf-topology feature installed?</em></b></p> <p>Sample apis:</p> <p><b>GET</b>: curl -v GET -u admin:admin http://&lt;ip&gt;:8181/restconf/operational/network-topology:network-topology/topology/topology-netconf/node/&lt;device_name&gt;/yang-ext:mount</p> <p><b>PUT</b>: curl -H 'Content-Type: application/json' -H 'Accept: application/json' -v -X PUT -T policy-options.json -u admin:admin <a href="http://10.100.100.105:30181/restconf/config/network-topology:network-topology/topology/topology-netconf/node/18579-sim-device/yang-ext:mount/junos-conf-root:configuration/junos-conf-policy-options:policy-options" class="external-link" target="_blank" rel="nofollow noopener">http://&lt;ip&gt;:8181/restconf/config/network-topology:network-topology/topology/topology-netconf/node/&lt;device_name&gt;/yang-ext:mount/junos-conf-root:configuration/junos-conf-policy-options:policy-options</a></p> <p> </p> <p>Right, and I would expect single-node deployments to not be suspectible to timing (quoted 5000ms). Distributed systems are pesky <img class="emoticon" src="https://jira.opendaylight.org/images/icons/emoticons/sad.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>In a cluster, odl-netconf-topology has a snowflake's chance in hell to work – the three instances will end up tripping over each other.</p> <p>But, if that works okay, it is a matter of clustering integration and we need to understand what is going wrong. Unfortunately I cannot look at this closely enough because of other (long overdue) stuff...</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=rovarga" class="user-hover" rel="rovarga">rovarga</a> In the above point, u r suggesting we could check and enhance "odl-netconf-topology" to work as a cluster to create "Mount points" in all of the ODL instance?</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=rovarga" class="user-hover" rel="rovarga">rovarga</a> I hope I kind of have figured out the root cause for the above mount stuck issue, </p> <p><b>Reason</b>: Singleton actor <span class="error">&#91;akka://opendaylight-cluster-data/system/singletonManagerOwnerSupervisor/OwnerSupervisor&#93;</span> was terminated</p> <p>And this actor doesn't resumes. Ideally the ClusterSingletonManager should make sure that this Single actor always runs right? Even if it terminates it should be restarted I think. </p> <p> </p> <p>I could understand controller/<b>EOSMain.java</b> is the one creating this ClusterSingleton actor,</p> <p>// start the initial sync behavior that switches to the regular one after syncing<br/> ownerSupervisor = clusterSingleton.init(<br/> SingletonActor.of(IdleSupervisor.create(iidCodec), "OwnerSupervisor"));</p> <p> </p> <p>And the class of this actor is controller/<b>OwnerSupervisor.java</b>, </p> <p>as this actor is terminated, the DOMEntities are stuck in a REMOTE_OWNERSHIP_LOST_NO_OWNER stage and no odl instance takes the master ownership (LOCAL_OWNERSHIP_GRANTED) and proceeds with the mount.</p> <p> </p> <p>If we do restart the ODL instance, then this Singleton OwnerSupervisor is getting started in any of the other ODL instance and that resumes the on-hold mounts. </p> <p> </p> <p>Could see akka.cluster.singleton.hand-over-retry-interval = 1s is configured in factory/akka.conf, but still not sure why the OwnerSupervisor actor doesn't restarts in the other instance when it gets terminated.</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=rovarga" class="user-hover" rel="rovarga">rovarga</a> Found a solution for the above issue, when increased the below "distributed-data" intervals from ms to 5s in akka.conf, we were able to mount 250 netconf devices at the same time to the ODL.</p> <p> </p> <p>akka.cluster.distributed-data.gossip-interval = 5s</p> <p>akka.cluster.distributed-data.notify-subscribers-interval = 5s</p> <p> </p> <p>akka {</p> <p>     cluster {</p> <p>        distributed-data </p> {         # How often the Replicator should send out gossip information.         # This value controls how quickly Entity Ownership Service data is replicated         # across cluster nodes.         gossip-interval = 5s         # How often the subscribers will be notified of changes, if any.         # This value controls how quickly Entity Ownership Service decisions are         # propagated within a node.         notify-subscribers-interval = 5s       } <p> </p> <p>Added fix for restarting the EOS Singleton Actor - <a href="https://jira.opendaylight.org/browse/CONTROLLER-2035" title="ODL Cluster - Akka Cluster Singleton Manager Actor - OwnerSupervisor Getting Terminated and never restarts" class="issue-link" data-issue-key="CONTROLLER-2035"><del>CONTROLLER-2035</del></a> ODL Cluster - Akka Cluster Singleton Manager Actor - OwnerSupervisor Getting Terminated and never restarts - OpenDaylight JIRA</p> <p> </p> Development Rank 0|hzzzxw: