Build Version: Phosphorus SR1

3 Node Cluster

From the Logs - member_1 is chosen as the Master for the Devices 

Scenario:

--------------------

Triggered 200 Mount requests at the same time to ODL.

In the logs, could see the requests are received and the Mount process was started for few of the devices, but after sometime we start getting the AskTimeoutException continuously which causes the further device mounts.

When checked the mount status only around 78 devices among 200 were Connected, and other devices mount got stuck due to above, it dint even span the Netconf actor for the mounting. 

2022-02-22T20:43:22,747 | ERROR | opendaylight-cluster-data-akka.actor.default-dispatcher-46 | SupervisorStrategy               | 213 - org.opendaylight.controller.repackaged-akka - 4.0.7 | Ask timed out on Actor[akka://opendaylight-cluster-data/system/typedDdataReplicator#-1603400317]] after [5000 ms]. Message of type [akka.cluster.ddata.typed.javadsl.Replicator$Get]. A typical reason for `AskTimeoutException` is that the recipient actor didn't send a reply.2022-02-22T20:43:22,747 | ERROR | opendaylight-cluster-data-akka.actor.default-dispatcher-46 | SupervisorStrategy               | 213 - org.opendaylight.controller.repackaged-akka - 4.0.7 | Ask timed out on Actor[akka://opendaylight-cluster-data/system/typedDdataReplicator#-1603400317]] after [5000 ms]. Message of type [akka.cluster.ddata.typed.javadsl.Replicator$Get]. A typical reason for `AskTimeoutException` is that the recipient actor didn't send a reply.java.util.concurrent.TimeoutException: Ask timed out on Actor[akka://opendaylight-cluster-data/system/typedDdataReplicator#-1603400317]] after [5000 ms]. Message of type [akka.cluster.ddata.typed.javadsl.Replicator$Get]. A typical reason for `AskTimeoutException` is that the recipient actor didn't send a reply. at akka.actor.typed.scaladsl.AskPattern$.$anonfun$onTimeout$1(AskPattern.scala:131) ~[bundleFile:?] at akka.pattern.PromiseActorRef$.$anonfun$apply$1(AskSupport.scala:730) ~[bundleFile:?] at akka.actor.Scheduler$$anon$7.run(Scheduler.scala:479) ~[bundleFile:?] at scala.concurrent.ExecutionContext$parasitic$.execute(ExecutionContext.scala:222) ~[bundleFile:?] at akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:365) ~[bundleFile:?] at akka.actor.LightArrayRevolverScheduler$$anon$3.executeBucket$1(LightArrayRevolverScheduler.scala:314) ~[bundleFile:?] at akka.actor.LightArrayRevolverScheduler$$anon$3.nextTick(LightArrayRevolverScheduler.scala:318) ~[bundleFile:?] at akka.actor.LightArrayRevolverScheduler$$anon$3.run(LightArrayRevolverScheduler.scala:270) ~[bundleFile:?] at java.lang.Thread.run(Thread.java:829) ~[?:?]2022-02-22T20:43:22,788 | INFO  | opendaylight-cluster-data-akka.actor.default-dispatcher-18 | RemoteActorRefProvider$RemoteDeadLetterActorRef | 213 - org.opendaylight.controller.repackaged-akka - 4.0.7 | Message [akka.cluster.ddata.typed.javadsl.Replicator$GetSuccess] to Actor[akka://opendaylight-cluster-data/deadLetters] was not delivered. [5] dead letters encountered. If this is not an expected behavior then Actor[akka://opendaylight-cluster-data/deadLetters] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.2022-02-22T20:43:22,726 | ERROR | opendaylight-cluster-data-akka.actor.default-dispatcher-15 | Behavior$                        | 213 - org.opendaylight.controller.repackaged-akka - 4.0.7 | Supervisor StopSupervisor saw failure: Ask timed out on Actor[akka://opendaylight-cluster-data/system/typedDdataReplicator#-1603400317]] after [5000 ms]. Message of type [akka.cluster.ddata.typed.javadsl.Replicator$Update]. A typical reason for `AskTimeoutException` is that the recipient actor didn't send a reply.java.util.concurrent.TimeoutException: Ask timed out on Actor[akka://opendaylight-cluster-data/system/typedDdataReplicator#-1603400317]] after [5000 ms]. Message of type [akka.cluster.ddata.typed.javadsl.Replicator$Update]. A typical reason for `AskTimeoutException` is that the recipient actor didn't send a reply. at akka.actor.typed.scaladsl.AskPattern$.$anonfun$onTimeout$1(AskPattern.scala:131) ~[bundleFile:?] at akka.pattern.PromiseActorRef$.$anonfun$apply$1(AskSupport.scala:730) ~[bundleFile:?] at akka.actor.Scheduler$$anon$7.run(Scheduler.scala:479) ~[bundleFile:?] at scala.concurrent.ExecutionContext$parasitic$.execute(ExecutionContext.scala:222) ~[bundleFile:?] at akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:365) ~[bundleFile:?] at akka.actor.LightArrayRevolverScheduler$$anon$3.executeBucket$1(LightArrayRevolverScheduler.scala:314) ~[bundleFile:?] at akka.actor.LightArrayRevolverScheduler$$anon$3.nextTick(LightArrayRevolverScheduler.scala:318) ~[bundleFile:?] at akka.actor.LightArrayRevolverScheduler$$anon$3.run(LightArrayRevolverScheduler.scala:270) ~[bundleFile:?] at java.lang.Thread.run(Thread.java:829) ~[?:?]2022-02-22T20:43:22,805 | INFO  | opendaylight-cluster-data-akka.actor.default-dispatcher-16 | ClusterSingletonManager          | 213 - org.opendaylight.controller.repackaged-akka - 4.0.7 | Singleton actor [akka://opendaylight-cluster-data/system/singletonManagerOwnerSupervisor/OwnerSupervisor] was terminated2022-02-22T20:43:22,806 | INFO  | opendaylight-cluster-data-akka.actor.default-dispatcher-16 | LocalActorRef                    | 213 - org.opendaylight.controller.repackaged-akka - 4.0.7 | Message [akka.cluster.ddata.typed.javadsl.Replicator$Changed] wrapped in [akka.actor.typed.internal.AdaptMessage] to Actorakka://opendaylight-cluster-data/system/singletonManagerOwnerSupervisor/OwnerSupervisor#901657858 was not delivered. [6] dead letters encountered. If this is not an expected behavior then Actorakka://opendaylight-cluster-data/system/singletonManagerOwnerSupervisor/OwnerSupervisor#901657858 may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.2022-02-22T20:43:22,807 | INFO  | opendaylight-cluster-data-akka.actor.default-dispatcher-16 | LocalActorRef                    | 213 - org.opendaylight.controller.repackaged-akka - 4.0.7 | Message [akka.cluster.ddata.typed.javadsl.Replicator$Changed] wrapped in [akka.actor.typed.internal.AdaptMessage] to Actorakka://opendaylight-cluster-data/system/singletonManagerOwnerSupervisor/OwnerSupervisor#901657858 was not delivered. [7] dead letters encountered. If this is not an expected behavior then Actorakka://opendaylight-cluster-data/system/singletonManagerOwnerSupervisor/OwnerSupervisor#901657858 may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.