https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/CONTROLLER-1546 controller <p>Image USed RC3.3: <a href="https://nexus.opendaylight.org/content/repositories/autorelease-1484/org/opendaylight/integration/distribution-karaf/0.5.0-Boron/distribution-karaf-0.5.0-Boron.zip" class="external-link" target="_blank" rel="nofollow noopener">https://nexus.opendaylight.org/content/repositories/autorelease-1484/org/opendaylight/integration/distribution-karaf/0.5.0-Boron/distribution-karaf-0.5.0-Boron.zip</a></p> <p>Operations:</p> <p> 1. Setup 3 node ODL (feature: odl-ovsdb-openstack aka Legacy Netvirt)</p> <p> 2. Attempted failover in thie sequence</p> <p> ODL1 down ODL2 up ODL3 up</p> <p> ODL1 up, ODL2 down ODL3 up</p> <p> --&gt; Operations started to fail here</p> <p> ODL1 up, ODL2 up, ODL3 up</p> <p> --&gt; no recovery </p> <p> At this point ODL3 was listed as "owner" for all entities. All the operations were failing. Checking the logs in ODL2 and ODL3 indiciated a lot of blueprint errors and some WARN indicating problems with remoterpc</p> <p>2016-09-09 21:48:21,009 | WARN | ult-dispatcher-2 | RpcRegistry | 168 - org.opendaylight.controller.sal-remoterpc-connector - 1.4.0.Boron | Timed out finding routers for RouteIdentifierImpl{context=null, type=(urn:opendaylight:packet:service?revision=2013-07-09)transmit-packet, route=/(urn:opendaylight:inventory?revision=2013-08-19)nodes/node/node[</p> {(urn:opendaylight:inventory?revision=2013-08-19)id=openflow:92945849353687} <p>]}<br/> 2016-09-09 21:48:21,009 | WARN | ult-dispatcher-2 | RpcRegistry | 168 - org.opendaylight.controller.sal-remoterpc-connector - 1.4.0.Boron | Timed out finding routers for RouteIdentifierImpl{context=null, type=(urn:opendaylight:packet:service?revision=2013-07-09)transmit-packet, route=/(urn:opendaylight:inventory?revision=2013-08-19)nodes/node/node[</p> {(urn:opendaylight:inventory?revision=2013-08-19)id=openflow:92945849353687} <p>]}<br/> 2016-09-09 21:48:21,009 | WARN | ult-dispatcher-2 | RpcRegistry | 168 - org.opendaylight.controller.sal-remoterpc-connector - 1.4.0.Boron | Timed out finding routers for RouteIdentifierImpl{context=null, type=(urn:opendaylight:packet:service?revision=2013-07-09)transmit-packet, route=/(urn:opendaylight:inventory?revision=2013-08-19)nodes/node/node[</p> {(urn:opendaylight:inventory?revision=2013-08-19)id=openflow:92945849353687} <p>]}</p> <p>Operating System: All<br/> Platform: All</p> CONTROLLER-1546

Operations Failed after Failover with exceptions and Errors

Bug Resolved Cannot Reproduce Unassigned Venkatrangan Govindarajan Fri, 9 Sep 2016 21:48:41 +0000 Tue, 25 Jul 2023 08:24:12 +0000 Tue, 11 Oct 2016 20:53:24 +0000 clustering 0 9 <p>Attachment odl1_log.tgz has been added with description: ODL1 Logs</p> <p>Attachment odl2_log.tgz has been added with description: ODL2 logs</p> <p>Attachment odl3_log.tgz has been added with description: ODL3 logs</p> <p>Logs uploaded, Please check the blueprint errors in ODL2 and remoteprc errors in ODL3.</p> <p>ODL1 log has some entrie with "NullPointer Exception" while startup mostly when invoking entity ownership API.</p> <p>Could not reproduce the failure. But still the ERROR and exceptins seen in log</p> <p>There are many NPE's in ForwardingRulesManagerImpl:</p> <p>2016-09-09 20:39:10,864 | ERROR | on-dispatcher-31 | DataTreeChangeListenerActor | 170 - org.opendaylight.controller.sal-distributed-datastore - 1.4.0.Boron | Error notifying listener org.opendaylight.controller.md.sal.binding.impl.BindingClusteredDOMDataTreeChangeListenerAdapter@2a4f203b<br/> java.lang.NullPointerException<br/> at org.opendaylight.openflowplugin.applications.frm.impl.ForwardingRulesManagerImpl.isNodeOwner(ForwardingRulesManagerImpl.java:261)<span class="error">&#91;281:org.opendaylight.openflowplugin.applications.forwardingrules-manager:0.3.0.Boron&#93;</span><br/> at org.opendaylight.openflowplugin.applications.frm.impl.AbstractListeningCommiter.preConfigurationCheck(AbstractListeningCommiter.java:111)<span class="error">&#91;281:org.opendaylight.openflowplugin.applications.forwardingrules-manager:0.3.0.Boron&#93;</span><br/> at org.opendaylight.openflowplugin.applications.frm.impl.AbstractListeningCommiter.onDataTreeChanged(AbstractListeningCommiter.java:53)<span class="error">&#91;281:org.opendaylight.openflowplugin.applications.forwardingrules-manager:0.3.0.Boron&#93;</span><br/> at org.opendaylight.controller.md.sal.binding.impl.BindingDOMDataTreeChangeListenerAdapter.onDataTreeChanged(BindingDOMDataTreeChangeListenerAdapter.java:41)<span class="error">&#91;141:org.opendaylight.controller.sal-binding-broker-impl:1.4.0.Boron&#93;</span><br/> at org.opendaylight.controller.cluster.datastore.DataTreeChangeListenerActor.dataChanged(DataTreeChangeListenerActor.java:55)<span class="error">&#91;170:org.opendaylight.controller.sal-distributed-datastore:1.4.0.Boron&#93;</span></p> <p>Also a few in NeutronNetworkChangeListener:</p> <p>2016-09-09 20:39:12,906 | ERROR | on-dispatcher-38 | DataChangeListener | 170 - org.opendaylight.controller.sal-distributed-datastore - 1.4.0.Boron | Error notifying listener org.opendaylight.netvirt.openstack.netvirt.translator.iaware.impl.NeutronNetworkChangeListener<br/> java.lang.NullPointerException<br/> at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:210)<span class="error">&#91;38:com.google.guava:18.0.0&#93;</span><br/> at org.opendaylight.netvirt.openstack.netvirt.AbstractHandler.enqueueEvent(AbstractHandler.java:69)<span class="error">&#91;290:org.opendaylight.netvirt.openstack.net-virt:1.3.0.Boron&#93;</span><br/> at org.opendaylight.netvirt.openstack.netvirt.NetworkHandler.neutronNetworkCreated(NetworkHandler.java:68)<span class="error">&#91;290:org.opendaylight.netvirt.openstack.net-virt:1.3.0.Boron&#93;</span><br/> at org.opendaylight.netvirt.openstack.netvirt.translator.iaware.impl.NeutronNetworkChangeListener.createNetwork(NeutronNetworkChangeListener.java:87)<span class="error">&#91;290:org.opendaylight.netvirt.openstack.net-virt:1.3.0.Boron&#93;</span><br/> at org.opendaylight.netvirt.openstack.netvirt.translator.iaware.impl.NeutronNetworkChangeListener.onDataChanged(NeutronNetworkChangeListener.java:74)<span class="error">&#91;290:org.opendaylight.netvirt.openstack.net-virt:1.3.0.Boron&#93;</span><br/> at org.opendaylight.controller.md.sal.binding.impl.AbstractForwardedDataBroker$TranslatingDataChangeInvoker.onDataChanged(AbstractForwardedDataBroker.java:143)<span class="error">&#91;141:org.opendaylight.controller.sal-binding-broker-impl:1.4.0.Boron&#93;</span><br/> at org.opendaylight.controller.cluster.datastore.DataChangeListener.dataChanged(DataChangeListener.java:71)<span class="error">&#91;170:org.opendaylight.controller.sal-distributed-datastore:1.4.0.Boron&#93;</span></p> <p>These should be looked at by someone familiar with that code. I don't know the impact.</p> <p>On ODL3, I see a couple transaction failures and a read failure ("Failure to delete ovsdbNode") around 2016-09-09 18:15:15 with message "Metadata not available" which indicates it tried to delete/read a node that doesn't exist.</p> <p>I don't see any blueprint errors on ODL2 but any blueprint issues would occur on startup.</p> <p>The RpcRegistry warning means a client tried to send a routed RPC but there's no implementation registered for the routed node path. That could be b/c there was a prior registration but was unregistered or there never was/is a registration or there is a registration on a remote controller node but hasn't propagated to the calling controller node yet. However there is a 5 sec wait for convergence hence the "Timed out finding routers" message. </p> <p>Wrt the EOS, member-2 became the shard leader originally and transferred leadership to member-3 when it was shut down at 2016-09-09 20:47:54. Since both member-1 and member-2 had been shut down and restarted, I would expect for member-3 would be the owner for the entities. Interestingly, 2016-09-09 20:13:01,009 is the last timestamp in the logs for ODL3.</p> <p>I'm unclear as to the exact issue observed as I'm not familiar with ovsdb. I'm also unclear as to whether there's any issue with clustering here. As I mentioned the EOS behavior looks correct but w/o EOS debug enabled I can't tell for sure.</p> <p>The NPE's mentioned above could be significant. I would suggest having ovsdb folks take a look.</p> <p>From the logs, the sequence of restarts is:</p> <p>2016-09-09 18:13 - all nodes up<br/> 2016-09-09 18:41 - ODL1 stopped<br/> 2016-09-09 20:38 - ODL1 restarted<br/> 2016-09-09 20:47 - ODL2 stopped<br/> 2016-09-09 21:15 - ODL2 restarted</p> <p>Followed the steps mentioned above. <br/> Observed the karaf logs, but couldn't find the above warn messages.<br/> Thus, the bug is not reproduced.</p> <p>Closing this as it's not reproducible and whatever issue there was doesn't seem to related to clustering based on analysis of the logs. As mentioned before, the NPE's emanating from ovsdb look ominous and may have been the cause so I would suggest creating a bug in that project.</p> Development External issue ID 6686 External issue URL Rank 0|i02r8n: