Details
-
Bug
-
Status: Resolved
-
Highest
-
Resolution: Cannot Reproduce
-
None
-
None
-
None
Description
In a 3 node Cluster, ODL Version: Silicon SR4
features Enabled: odl-restconf-all and odl-netconf-clustered-topology
Scenario:
Tried to mount 250 netconf devices via mount POST restApi, each mount api send synchronously with a time gap of around 2 secs between each apis.
After around 40-50 passes APIs, all the other mount POST apis gets failed with the error,
Shard Has no Leader or AskTimeoutException
2022-02-08T21:06:07,310 | INFO | qtp1545674178-437 | RestconfImpl | 279 - org.opendaylight.netconf.restconf-nb-bierman02 - 1.13.8 | Error creating data config/network-topology:network-topology/topology/topology-netconf
2022-02-08T21:06:07,310 | INFO | qtp1545674178-437 | RestconfImpl | 279 - org.opendaylight.netconf.restconf-nb-bierman02 - 1.13.8 | Error creating data config/network-topology:network-topology/topology/topology-netconfjava.util.concurrent.ExecutionException: TransactionCommitFailedException{message=canCommit encountered an unexpected failure, errorList=[RpcError [message=canCommit encountered an unexpected failure, severity=ERROR, errorType=APPLICATION, tag=operation-failed, applicationTag=null, info=null, cause=org.opendaylight.mdsal.common.api.DataStoreUnavailableException: Could not process forwarded ready transaction member-1-datastore-config-fe-0-txn-28-0. Shard member-3-shard-topology-config currently has no leader. Try again later.]]}
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:564) ~[bundleFile:?]
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:545) ~[bundleFile:?]
at com.google.common.util.concurrent.FluentFuture$TrustedFuture.get(FluentFuture.java:88) ~[bundleFile:?]
at org.opendaylight.netconf.sal.restconf.impl.RestconfImpl.createConfigurationData(RestconfImpl.java:966) ~[bundleFile:?]
at org.opendaylight.netconf.sal.restconf.impl.RestconfImpl.createConfigurationData(RestconfImpl.java:906) ~[bundleFile:?]
at org.opendaylight.netconf.sal.restconf.impl.StatisticsRestconfServiceWrapper.createConfigurationData(StatisticsRestconfServiceWrapper.java:161) ~[bundleFile:?]
at org.opendaylight.netconf.sal.rest.impl.RestconfCompositeWrapper.createConfigurationData(RestconfCompositeWrapper.java:86) ~[bundleFile:?]
at jdk.internal.reflect.GeneratedMethodAccessor72.invoke(Unknown Source) ~[?:?]
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
2022-02-11T13:15:23,853 | INFO | qtp1228907817-891 | RestconfImpl | 279 - org.opendaylight.netconf.restconf-nb-bierman02 - 1.13.8 | Error creating data config/network-topology:network-topology/topology/topology-netconfjava.util.concurrent.ExecutionException: TransactionCommitFailedException{message=canCommit encountered an unexpected failure, errorList=[RpcError [message=canCommit encountered an unexpected failure, severity=ERROR, errorType=APPLICATION, tag=operation-failed, applicationTag=null, info=null, cause=akka.pattern.AskTimeoutException: Ask timed out on ActorSelection[Anchor(akka://opendaylight-cluster-data/), Path(/user/shardmanager-config/member-1-shard-topology-config#-1826243286)] after [30000 ms]. Message of type [org.opendaylight.controller.cluster.datastore.messages.ReadyLocalTransaction]. A typical reason for `AskTimeoutException` is that the recipient actor didn't send a reply.]]}
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:564) ~[bundleFile:?]
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:545) ~[bundleFile:?]
at com.google.common.util.concurrent.FluentFuture$TrustedFuture.get(FluentFuture.java:88) ~[bundleFile:?]
at org.opendaylight.netconf.sal.restconf.impl.RestconfImpl.createConfigurationData(RestconfImpl.java:966) ~[bundleFile:?]
at org.opendaylight.netconf.sal.restconf.impl.RestconfImpl.createConfigurationData(RestconfImpl.java:906) ~[bundleFile:?]
at org.opendaylight.netconf.sal.restconf.impl.StatisticsRestconfServiceWrapper.createConfigurationData(StatisticsRestconfServiceWrapper.java:161) ~[bundleFile:?]
at org.opendaylight.netconf.sal.rest.impl.RestconfCompositeWrapper.createConfigurationData(RestconfCompositeWrapper.java:86) ~[bundleFile:?]
at jdk.internal.reflect.GeneratedMethodAccessor80.invoke(Unknown Source) ~[?:?]
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
When checked on the Shards Role assigned:
Initially when the ODL is started, could see the member-1 is assigned the Leader and later after pushing the mount apis, could see reelection happening and the role changes from Leader to IsolatedLeader.