https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/OPNFLWPLUG-1028 OpenFlowPlugin <p>L2 suites in netvirt are failing randomly in 3 node, and in one such failed instance I noticed that Table Miss entry for Table 43 is not programmed on switch.</p> <p>I can see the flow being present in config/opendaylight-inventory, but flow is missing on the switch.</p> <p><a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/229/faseela-l2-netvirt-csit-3node-openstack-queens-upstream-stateful-fluorine/15/" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/229/faseela-l2-netvirt-csit-3node-openstack-queens-upstream-stateful-fluorine/15/</a></p> <p>Problematic node id - openflow:80006780316000</p> <p>Model Dumps can be found at the Tear Down Dumps of the first failing TC "Check If VMs got IP Address". Let me know if any more information is needed.</p> OPNFLWPLUG-1028

Table Miss Entry failed to program in 3 node netvirt CSIT

Bug Highest Resolved Done Somashekhar Javalagi Faseela K csit csit:3node Tue, 17 Jul 2018 06:43:52 +0000 Mon, 20 Aug 2018 04:57:59 +0000 Mon, 20 Aug 2018 04:57:59 +0000 Oxygen Fluorine clustering 0 8 <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=k.faseela" class="user-hover" rel="k.faseela">k.faseela</a>, could you try to find a log line in karaf.log (I guess it needs to be on the leader) for when<br/> that table=43 flow made it to config? We can use that timestamp to go check the ovs logs to see if<br/> there are any clues there? One possibility is that OFP took care of sending the flow down, but it was<br/> rejected for some reason. We can also use that timestamp to check all three karaf.logs for any clues.</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=jluhrsen" class="user-hover" rel="jluhrsen">jluhrsen</a> : This issue came only once out of 20 runs. With INFO in karaf logs, there is nothing to track to see when exactly ELAN pushed this flow. But I see that ELAN has pushed the flow by checking the same in the datastore dump.</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=Avishnoi" class="user-hover" rel="Avishnoi">Avishnoi</a> : Any one from ofp who can take this up? This issue is showing up in several of our 3 node CSIT runs.</p> <p>Another instance, where table miss flow in Table 50 was present initially, and all of a suddent went off :</p> <p> </p> <p><a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/254/faseela-l2-netvirt-csit-3node-0cmb-1ctl-2cmp-openstack-queens-upstream-stateful-oxygen/" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/254/faseela-l2-netvirt-csit-3node-0cmb-1ctl-2cmp-openstack-queens-upstream-stateful-oxygen/</a></p> <p> </p> <p>Could you please enable the following package logging and provide the logs?</p> <p> log:set DEBUG org.opendaylight.openflowplugin.impl</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=Arunprakash" class="user-hover" rel="Arunprakash">Arunprakash</a> : Here are the logs with DEBUG enabled. Table 18 entry is missing in this case. It took several runs to reproduce the issue when DEBUG was enabled :</p> <p> </p> <p><a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/260/faseela-l2-netvirt-csit-3node-0cmb-1ctl-2cmp-openstack-queens-upstream-stateful-oxygen/18/" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/260/faseela-l2-netvirt-csit-3node-0cmb-1ctl-2cmp-openstack-queens-upstream-stateful-oxygen/18/</a></p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=k.faseela" class="user-hover" rel="k.faseela">k.faseela</a> do we want to assign this to <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=Arunprakash" class="user-hover" rel="Arunprakash">Arunprakash</a>? thanks for looking at this guys...</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=Arunprakash" class="user-hover" rel="Arunprakash">Arunprakash</a> will analyze the logs and let us know whom to assign the bug to. </p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=d.arunprakash%40ericsson.com" class="user-hover" rel="d.arunprakash@ericsson.com">d.arunprakash@ericsson.com</a> : Taking at the logs, i can see the below log in <a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/260/faseela-l2-netvirt-csit-3node-0cmb-1ctl-2cmp-openstack-queens-upstream-stateful-oxygen/18/odl_1/odl1_karaf.log.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/260/faseela-l2-netvirt-csit-3node-0cmb-1ctl-2cmp-openstack-queens-upstream-stateful-oxygen/18/odl_1/odl1_karaf.log.gz</a> 10 mins before the test suite starts :</p> <p>2018-07-25T23:00:41,688 | INFO | pool-241-thread-1 | WeightedCentralizedSwitchScheduler | 361 - org.opendaylight.netvirt.natservice-impl - 0.6.3.SNAPSHOT | addSwitch: Adding 167740445697660 dpnId to switchWeightsMap</p> <p>2018-07-25T23:00:42,274 | DEBUG | epollEventLoopGroup-9-1 | SalFlowServiceImpl | 385 - org.opendaylight.openflowplugin.impl - 0.6.3.SNAPSHOT | Flow add with id=DHCPTableMissFlowForExternalTunnel finished without error</p> <p>But the flow is not present on the switch when the suite starts.</p> <p>We have 3 OVS in the test, but i see only this flow-id log for 2 switches. Anyways the compute in question, is  167740445697660, so let us assume that the flow was getting pushed for the same, but doesn't show up on switch. But there are different failures showing up in the logs including the below one. Let me know if this could impact:</p> <p> </p> <p>2018-07-25T23:01:09,328 | ERROR | opendaylight-cluster-data-notification-dispatcher-97 | DefaultConfigPusher | 378 - org.opendaylight.openflowplugin.applications.of-switch-config-pusher - 0.6.3.SNAPSHOT | Future (eventually) failed: addFlow org.opendaylight.controller.md.sal.dom.api.DOMRpcImplementationNotAvailableException: No implementation of RPC AbsoluteSchemaPath{path=<span class="error">&#91;(urn:opendaylight:module:config?revision=2014-10-15)set-config&#93;</span>} available at org.opendaylight.controller.md.sal.dom.broker.impl.RoutedDOMRpcRoutingTableEntry.invokeRpc(RoutedDOMRpcRoutingTableEntry.java:85) <span class="error">&#91;229:org.opendaylight.controller.sal-broker-impl:1.7.3.SNAPSHOT&#93;</span> at org.opendaylight.controller.md.sal.dom.broker.impl.DOMRpcRoutingTable.invokeRpc(DOMRpcRoutingTable.java:178) <span class="error">&#91;229:org.opendaylight.controller.sal-broker-impl:1.7.3.SNAPSHOT&#93;</span> at org.opendaylight.controller.md.sal.dom.broker.impl.DOMRpcRouter.invokeRpc(DOMRpcRouter.java:102) <span class="error">&#91;229:org.opendaylight.controller.sal-broker-impl:1.7.3.SNAPSHOT&#93;</span> at Proxybb8ecbf1_eb9d_4480_a18e_4a2007f00a3c.invokeRpc(Unknown Source) <span class="error">&#91;?:?&#93;</span> at Proxy40f45215_a726_4be9_ac6b_649862163662.invokeRpc(Unknown Source) <span class="error">&#91;?:?&#93;</span> at org.opendaylight.controller.md.sal.binding.impl.RpcServiceAdapter.invoke0(RpcServiceAdapter.java:68) <span class="error">&#91;226:org.opendaylight.controller.sal-binding-broker-impl:1.7.3.SNAPSHOT&#93;</span> at org.opendaylight.controller.md.sal.binding.impl.RpcServiceAdapter.access$000(RpcServiceAdapter.java:46) <span class="error">&#91;226:org.opendaylight.controller.sal-binding-broker-impl:1.7.3.SNAPSHOT&#93;</span> at org.opendaylight.controller.md.sal.binding.impl.RpcServiceAdapter$RpcInvocationStrategy.invoke(RpcServiceAdapter.java:165) <span class="error">&#91;226:org.opendaylight.controller.sal-binding-broker-impl:1.7.3.SNAPSHOT&#93;</span> at org.opendaylight.controller.md.sal.binding.impl.RpcServiceAdapter.invoke(RpcServiceAdapter.java:99) <span class="error">&#91;226:org.opendaylight.controller.sal-binding-broker-impl:1.7.3.SNAPSHOT&#93;</span> at com.sun.proxy.$Proxy113.setConfig(Unknown Source) <span class="error">&#91;388:org.opendaylight.openflowplugin.model.flow-service:0.6.3.SNAPSHOT&#93;</span> at org.opendaylight.openflowplugin.openflow.ofswitch.config.DefaultConfigPusher.onDataTreeChanged(DefaultConfigPusher.java:84) <span class="error">&#91;378:org.opendaylight.openflowplugin.applications.of-switch-config-pusher:0.6.3.SNAPSHOT&#93;</span> at org.opendaylight.controller.md.sal.binding.impl.BindingDOMDataTreeChangeListenerAdapter.onDataTreeChanged(BindingDOMDataTreeChangeListenerAdapter.java:41) <span class="error">&#91;226:org.opendaylight.controller.sal-binding-broker-impl:1.7.3.SNAPSHOT&#93;</span> at org.opendaylight.controller.cluster.datastore.DataTreeChangeListenerActor.dataChanged(DataTreeChangeListenerActor.java:67) <span class="error">&#91;239:org.opendaylight.controller.sal-distributed-datastore:1.7.3.SNAPSHOT&#93;</span> at org.opendaylight.controller.cluster.datastore.DataTreeChangeListenerActor.handleReceive(DataTreeChangeListenerActor.java:41) <span class="error">&#91;239:org.opendaylight.controller.sal-distributed-datastore:1.7.3.SNAPSHOT&#93;</span> at org.opendaylight.controller.cluster.common.actor.AbstractUntypedActor.onReceive(AbstractUntypedActor.java:38) <span class="error">&#91;232:org.opendaylight.controller.sal-clustering-commons:1.7.3.SNAPSHOT&#93;</span> at akka.actor.UntypedActor$$anonfun$receive$1.applyOrElse(UntypedActor.scala:167) <span class="error">&#91;42:com.typesafe.akka.actor:2.5.11&#93;</span> at akka.actor.Actor.aroundReceive(Actor.scala:517) <span class="error">&#91;42:com.typesafe.akka.actor:2.5.11&#93;</span> at akka.actor.Actor.aroundReceive$(Actor.scala:515) <span class="error">&#91;42:com.typesafe.akka.actor:2.5.11&#93;</span> at akka.actor.UntypedActor.aroundReceive(UntypedActor.scala:97) <span class="error">&#91;42:com.typesafe.akka.actor:2.5.11&#93;</span> at akka.actor.ActorCell.receiveMessage(ActorCell.scala:590) <span class="error">&#91;42:com.typesafe.akka.actor:2.5.11&#93;</span> at akka.actor.ActorCell.invoke(ActorCell.scala:559) <span class="error">&#91;42:com.typesafe.akka.actor:2.5.11&#93;</span> at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257) <span class="error">&#91;42:com.typesafe.akka.actor:2.5.11&#93;</span> at akka.dispatch.Mailbox.run(Mailbox.scala:224) <span class="error">&#91;42:com.typesafe.akka.actor:2.5.11&#93;</span> at akka.dispatch.Mailbox.exec(Mailbox.scala:234) <span class="error">&#91;42:com.typesafe.akka.actor:2.5.11&#93;</span> at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) <span class="error">&#91;42:com.typesafe.akka.actor:2.5.11&#93;</span> at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) <span class="error">&#91;42:com.typesafe.akka.actor:2.5.11&#93;</span> at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) <span class="error">&#91;42:com.typesafe.akka.actor:2.5.11&#93;</span> at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) <span class="error">&#91;42:com.typesafe.akka.actor:2.5.11&#93;</span></p> <p>2018-07-25T23:01:00,416 | INFO | pool-93-thread-1 | SalRoleServiceImpl | 385 - org.opendaylight.openflowplugin.impl - 0.6.3.SNAPSHOT | SetRole called with input:SetRoleInput [_controllerRole=BECOMESLAVE, _node=NodeRef [_value=KeyedInstanceIdentifier{targetType=interface org.opendaylight.yang.gen.v1.urn.opendaylight.inventory.rev130819.nodes.Node, path=[org.opendaylight.yang.gen.v1.urn.opendaylight.inventory.rev130819.Nodes, org.opendaylight.yang.gen.v1.urn.opendaylight.inventory.rev130819.nodes.Node[key=NodeKey [_id=Uri <span class="error">&#91;_value=openflow:167740445697660&#93;</span>]]]}], augmentation=[]] 2018-07-25T23:01:00,417 | INFO | pool-93-thread-1 | SalRoleServiceImpl | 385 - org.opendaylight.openflowplugin.impl - 0.6.3.SNAPSHOT | Requesting state change to BECOMESLAVE 2018-07-25T23:01:00,417 | INFO | pool-93-thread-1 | SalRoleServiceImpl | 385 - org.opendaylight.openflowplugin.impl - 0.6.3.SNAPSHOT | RoleChangeTask called on device:openflow:167740445697660 OFPRole:BECOMESLAVE 2018-07-25T23:01:00,418 | INFO | pool-93-thread-1 | RoleService | 385 - org.opendaylight.openflowplugin.impl - 0.6.3.SNAPSHOT | getGenerationIdFromDevice called for device: openflow:167740445697660 2018-07-25T23:01:00,427 | WARN | pool-93-thread-1 | OutboundQueueEntry | 394 - org.opendaylight.openflowplugin.openflowjava.openflow-protocol-impl - 0.6.3.SNAPSHOT | Can't commit a completed message. 2018-07-25T23:01:00,428 | WARN | pool-93-thread-1 | HashedWheelTimer | 62 - io.netty.common - 4.1.22.Final | An exception was thrown by TimerTask. java.lang.NullPointerException: null at org.opendaylight.openflowplugin.impl.services.AbstractRequestCallback.onFailure(AbstractRequestCallback.java:72) ~<span class="error">&#91;?:?&#93;</span> at org.opendaylight.openflowjava.protocol.impl.core.connection.OutboundQueueEntry.commit(OutboundQueueEntry.java:57) ~<span class="error">&#91;394:org.opendaylight.openflowplugin.openflowjava.openflow-protocol-impl:0.6.3.SNAPSHOT&#93;</span> at org.opendaylight.openflowjava.protocol.impl.core.connection.StackedOutboundQueue.commitEntry(StackedOutboundQueue.java:36) ~<span class="error">&#91;394:org.opendaylight.openflowplugin.openflowjava.openflow-protocol-impl:0.6.3.SNAPSHOT&#93;</span> at org.opendaylight.openflowjava.protocol.impl.core.connection.AbstractStackedOutboundQueue.commitEntry(AbstractStackedOutboundQueue.java:61) ~<span class="error">&#91;394:org.opendaylight.openflowplugin.openflowjava.openflow-protocol-impl:0.6.3.SNAPSHOT&#93;</span> at org.opendaylight.openflowplugin.impl.connection.OutboundQueueProviderImpl.commitEntry(OutboundQueueProviderImpl.java:81) ~<span class="error">&#91;?:?&#93;</span> at org.opendaylight.openflowplugin.impl.services.AbstractService.handleServiceCall(AbstractService.java:165) ~<span class="error">&#91;?:?&#93;</span> at org.opendaylight.openflowplugin.impl.services.AbstractService.handleServiceCall(AbstractService.java:110) ~<span class="error">&#91;?:?&#93;</span> at org.opendaylight.openflowplugin.impl.services.RoleService.getGenerationIdFromDevice(RoleService.java:69) ~<span class="error">&#91;?:?&#93;</span> at org.opendaylight.openflowplugin.impl.services.sal.SalRoleServiceImpl.tryToChangeRole(SalRoleServiceImpl.java:84) ~<span class="error">&#91;?:?&#93;</span> at org.opendaylight.openflowplugin.impl.services.sal.SalRoleServiceImpl.setRole(SalRoleServiceImpl.java:78) ~<span class="error">&#91;?:?&#93;</span> at org.opendaylight.openflowplugin.impl.role.RoleContextImpl.sendRoleChangeToDevice(RoleContextImpl.java:165) ~<span class="error">&#91;?:?&#93;</span> at org.opendaylight.openflowplugin.impl.role.RoleContextImpl.makeDeviceSlave(RoleContextImpl.java:144) ~<span class="error">&#91;?:?&#93;</span> at org.opendaylight.openflowplugin.impl.role.RoleContextImpl.lambda$new$0(RoleContextImpl.java:70) ~<span class="error">&#91;?:?&#93;</span> at io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:663) <span class="error">&#91;62:io.netty.common:4.1.22.Final&#93;</span> at io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:738) <span class="error">&#91;62:io.netty.common:4.1.22.Final&#93;</span> at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:466) <span class="error">&#91;62:io.netty.common:4.1.22.Final&#93;</span> at java.lang.Thread.run(Thread.java:748) <span class="error">&#91;?:?&#93;</span> 2018-07-25T23:01:08,916 | INFO | nioEventLoopGroup-4-1 | LoggingHandler | 62 - io.netty.common - 4.1.22.Final | <span class="error">&#91;id: 0x8b315f38, L:/0:0:0:0:0:0:0:0:6640&#93;</span> READ: <span class="error">&#91;id: 0x0aaeab0b, L:/10.30.170.60:6640 - R:/10.30.170.84:60506&#93;</span></p> <p>Latest log run, with wireshark capture and inventory dumps:</p> <p><a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/285/faseela-l2-netvirt-csit-3node-0cmb-1ctl-2cmp-openstack-queens-upstream-stateful-oxygen/29/" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/285/faseela-l2-netvirt-csit-3node-0cmb-1ctl-2cmp-openstack-queens-upstream-stateful-oxygen/29/</a></p> <p> </p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=Somashekhar" class="user-hover" rel="Somashekhar">Somashekhar</a> : Here are the logs with DEBUGs enabled :</p> <p><a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/291/faseela-l2-netvirt-csit-3node-0cmb-1ctl-2cmp-openstack-queens-upstream-stateful-oxygen/32/robot-plugin/log_01_l2.html.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/291/faseela-l2-netvirt-csit-3node-0cmb-1ctl-2cmp-openstack-queens-upstream-stateful-oxygen/32/robot-plugin/log_01_l2.html.gz</a></p> <p> </p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=k.faseela" class="user-hover" rel="k.faseela">k.faseela</a>, I have added few logs for debugging, can you please run the csit with this.</p> <p><a href="https://git.opendaylight.org/gerrit/#/c/74717/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/74717/</a></p> <p>Here are the logs <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=Somashekhar" class="user-hover" rel="Somashekhar">Somashekhar</a> : </p> <p> </p> <p><a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/307/faseela-l2-netvirt-csit-3node-0cmb-1ctl-2cmp-openstack-queens-upstream-stateful-oxygen/39/" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/307/faseela-l2-netvirt-csit-3node-0cmb-1ctl-2cmp-openstack-queens-upstream-stateful-oxygen/39/</a></p> <p> </p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=k.faseela" class="user-hover" rel="k.faseela">k.faseela</a>, can you run the csit with FRM debug logs enabled and with Som's review?</p> <p> </p> <p>Package:</p> <p>org.opendaylight.openflowplugin.applications.frm</p> <p> </p> <p><a href="https://git.opendaylight.org/gerrit/#/c/74715/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/74715/</a></p> <p> </p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=Arunprakash" class="user-hover" rel="Arunprakash">Arunprakash</a>, <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=Somashekhar" class="user-hover" rel="Somashekhar">Somashekhar</a> : Running this right now.. Will update once we get a failure log.</p> <p> </p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=jluhrsen" class="user-hover" rel="jluhrsen">jluhrsen</a> : Do you think we should mark this as a blocker for Fluorine? This issue comes only in 3 node CSIT though.</p> <p>no, I don't think we are close enough with cluster stability to start marking CSIT failures in 3node jobs as blockers. especially sporadic ones. But, I'm crossing my fingers that we will be able to do this in Neon <img class="emoticon" src="https://jira.opendaylight.org/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>Found an issue in a single node job where table 45 did not have the default flow: <a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-0cmb-1ctl-2cmp-openstack-queens-upstream-stateful-fluorine/130/" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-0cmb-1ctl-2cmp-openstack-queens-upstream-stateful-fluorine/130/</a></p> <p>Openflowplugin-impl is deciding the mastership of the device and communicating same to the forwardingrules-manager through yang notification. But this yang notification is taking more time(around 15 to 25 msecond) to reach FRM, after which mastership will be set to the device. By the time yang notification reaches FRM, there will be flows start coming for the reconciliation. As the device is not yet mastered, these flows get rejected.</p> <p> </p> <p>There is no direct way to fix this, as the fix may involve lot of design changes.</p> <p>Are there any current workarounds? Understood that this is difficult but this is looking like a very bad issue. Seems like you can hit this at any time. Restarting isn't a good workaround though.</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=shague" class="user-hover" rel="shague">shague</a>, we are working on a  simplified fix which can be used for the current situation and move to a better solution later in neon.</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=Somashekhar" class="user-hover" rel="Somashekhar">Somashekhar</a> will raise a review tomorrow with the probable fix and <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=k.faseela" class="user-hover" rel="k.faseela">k.faseela</a> can run CSIT and confirm if it works.</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=k.faseela" class="user-hover" rel="k.faseela">k.faseela</a>, could you please provide the latest CSIT run logs, we want to verify the logs for the sequence of events.</p> <p><a href="https://git.opendaylight.org/gerrit/74715" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/74715</a></p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=k.faseela" class="user-hover" rel="k.faseela">k.faseela</a>, thanks for the link via email.</p> <p><a href="https://jenkins.opendaylight.org/sandbox/job/faseela-l2-netvirt-csit-3node-0cmb-1ctl-2cmp-openstack-queens-upstream-stateful-oxygen/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/sandbox/job/faseela-l2-netvirt-csit-3node-0cmb-1ctl-2cmp-openstack-queens-upstream-stateful-oxygen/</a></p> <p> </p> <p>We are able to check the sequence of events from openflowplugin to FRM and the owner is being set now before any default flows being programmed by applications.</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=Somashekhar" class="user-hover" rel="Somashekhar">Somashekhar</a>, you can work on formalizing the review.</p> Relates OPNFLWPLUG-1026 Development Epic Link NETVIRT-996 Rank 0|i03gs7: