https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/CONTROLLER-2029 controller <p>In a 3 node Cluster,  ODL Version: Silicon SR4</p> <p>features Enabled: odl-restconf-all and odl-netconf-clustered-topology</p> <p>Scenario:</p> <p>Tried to mount 250 netconf devices via mount POST restApi, each mount api send synchronously with a time gap of around 2 secs between each apis.</p> <p> </p> <p>After around 40-50 passes APIs, all the other mount POST apis gets failed with the error,</p> <p>Shard Has no Leader or AskTimeoutException</p> <p> </p> <p> </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>2022-02-08T21:06:07,310 | INFO  | qtp1545674178-437 | RestconfImpl                     | 279 - org.opendaylight.netconf.restconf-nb-bierman02 - 1.13.8 | Error creating data config/network-topology:network-topology/topology/topology-netconf 2022-02-08T21:06:07,310 | INFO  | qtp1545674178-437 | RestconfImpl                     | 279 - org.opendaylight.netconf.restconf-nb-bierman02 - 1.13.8 | Error creating data config/network-topology:network-topology/topology/topology-netconfjava.util.concurrent.ExecutionException: TransactionCommitFailedException{message=canCommit encountered an unexpected failure, errorList=[RpcError [message=canCommit encountered an unexpected failure, severity=ERROR, errorType=APPLICATION, tag=operation-failed, applicationTag=null, info=null, cause=org.opendaylight.mdsal.common.api.DataStoreUnavailableException: Could not process forwarded ready transaction member-1-datastore-config-fe-0-txn-28-0. Shard member-3-shard-topology-config currently has no leader. Try again later.]]} at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:564) ~[bundleFile:?] at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:545) ~[bundleFile:?] at com.google.common.util.concurrent.FluentFuture$TrustedFuture.get(FluentFuture.java:88) ~[bundleFile:?] at org.opendaylight.netconf.sal.restconf.impl.RestconfImpl.createConfigurationData(RestconfImpl.java:966) ~[bundleFile:?] at org.opendaylight.netconf.sal.restconf.impl.RestconfImpl.createConfigurationData(RestconfImpl.java:906) ~[bundleFile:?] at org.opendaylight.netconf.sal.restconf.impl.StatisticsRestconfServiceWrapper.createConfigurationData(StatisticsRestconfServiceWrapper.java:161) ~[bundleFile:?] at org.opendaylight.netconf.sal.rest.impl.RestconfCompositeWrapper.createConfigurationData(RestconfCompositeWrapper.java:86) ~[bundleFile:?] at jdk.internal.reflect.GeneratedMethodAccessor72.invoke(Unknown Source) ~[?:?] at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?] at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?] 2022-02-11T13:15:23,853 | INFO  | qtp1228907817-891 | RestconfImpl                     | 279 - org.opendaylight.netconf.restconf-nb-bierman02 - 1.13.8 | Error creating data config/network-topology:network-topology/topology/topology-netconfjava.util.concurrent.ExecutionException: TransactionCommitFailedException{message=canCommit encountered an unexpected failure, errorList=[RpcError [message=canCommit encountered an unexpected failure, severity=ERROR, errorType=APPLICATION, tag=operation-failed, applicationTag=null, info=null, cause=akka.pattern.AskTimeoutException: Ask timed out on ActorSelection[Anchor(akka://opendaylight-cluster-data/), Path(/user/shardmanager-config/member-1-shard-topology-config#-1826243286)] after [30000 ms]. Message of type [org.opendaylight.controller.cluster.datastore.messages.ReadyLocalTransaction]. A typical reason for `AskTimeoutException` is that the recipient actor didn't send a reply.]]} at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:564) ~[bundleFile:?] at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:545) ~[bundleFile:?] at com.google.common.util.concurrent.FluentFuture$TrustedFuture.get(FluentFuture.java:88) ~[bundleFile:?] at org.opendaylight.netconf.sal.restconf.impl.RestconfImpl.createConfigurationData(RestconfImpl.java:966) ~[bundleFile:?] at org.opendaylight.netconf.sal.restconf.impl.RestconfImpl.createConfigurationData(RestconfImpl.java:906) ~[bundleFile:?] at org.opendaylight.netconf.sal.restconf.impl.StatisticsRestconfServiceWrapper.createConfigurationData(StatisticsRestconfServiceWrapper.java:161) ~[bundleFile:?] at org.opendaylight.netconf.sal.rest.impl.RestconfCompositeWrapper.createConfigurationData(RestconfCompositeWrapper.java:86) ~[bundleFile:?] at jdk.internal.reflect.GeneratedMethodAccessor80.invoke(Unknown Source) ~[?:?] at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?] at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]</pre> </div></div> <p> </p> <p> </p> <p>When checked on the Shards Role assigned: </p> <p>Initially when the ODL is started, could see the member-1 is assigned the Leader and later after pushing the mount apis, could see reelection happening and the role changes from Leader to IsolatedLeader.</p> <p> </p> <p> </p> <p> </p> CONTROLLER-2029

ODL Clustering - Shard Has no Leader or AskTimeoutException Error - Silicon SR4

Bug Highest Resolved Cannot Reproduce Samuel Schneider Shibu Vijayakumar pt Fri, 11 Feb 2022 20:25:01 +0000 Wed, 26 Jul 2023 08:55:05 +0000 Thu, 18 May 2023 08:44:41 +0000 0 3 <p>Above issue not seen in ODL current release - Phosphorus SR1, but exists in Silicon SR3 and SR4 also.</p> <p>The difference seen reg the RoleChange is, in Silicon Version once the mount apis are pushed after some time, the Shards members are changing role, Leader changing to IsolatedLeader, but in Phosphorus not seeing those RoleChangeNotifier. Once after the intialization no rolechanges observed during the process of the mount apis.</p> <p>Well, the causes can be quite diverse and without karaf logs from all members I cannot determine the root cause.</p> <p>Without a root cause there is just no way I could pinpoint the difference.</p> <p>Attached the karaf logs of all the 3 members.</p> <p>I wasn't able to reproduce this issue on ODL Silicon SR4 3 node Cluster.<br/> I tried to connect 250 devices simulated by ODL-NETCONF testtool and all of them connected successfully.</p> <p>We would need some replicator test or consistent steps to reproduce for this issue.</p> <p>After looking trough the logs I noticed that all devices were connected by the same member-3, I could not reproduce this behavior. member-3 end up being without leader so this can be related to the issue.</p> <p>Also it seems bierman02 endpoint was used, which was removed in later versions of ODL. You can try using RFC-8040 endpoint instead.</p> Development Rank 0|i041jb: