https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/OPNFLWPLUG-760 OpenFlowPlugin <p>The latest regression runs for VTN CSIT jobs reports failure in normal flow installation when testing with OF 1.0 switches. The same jobs are not reporting failures when testing with OF 1.3 switches.</p> <p>VTN CSIT Jobs URL reporting failures<br/> <a href="https://jenkins.opendaylight.org/releng/view/vtn/job/vtn-csit-1node-manager-only-boron/637/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/vtn/job/vtn-csit-1node-manager-only-boron/637/</a><br/> <a href="https://jenkins.opendaylight.org/releng/view/vtn/job/vtn-csit-1node-coordinator-only-boron/209/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/vtn/job/vtn-csit-1node-coordinator-only-boron/209/</a></p> <p>Karaf Log URL<br/> <a href="https://logs.opendaylight.org/releng/jenkins092/vtn-csit-1node-manager-only-boron/637/archives/karaf.log.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/vtn-csit-1node-manager-only-boron/637/archives/karaf.log.gz</a><br/> <a href="https://logs.opendaylight.org/releng/jenkins092/vtn-csit-1node-coordinator-only-boron/209/archives/karaf.log.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/vtn-csit-1node-coordinator-only-boron/209/archives/karaf.log.gz</a></p> <p>Observation</p> <p>During the failure scenario the below error was observed in the log</p> <p>2016-08-28 19:34:22,691 | WARN | entLoopGroup-5-3 | DeviceContextImpl | 183 - org.opendaylight.openflowplugin.impl - 0.3.0.SNAPSHOT | Error processing port status message: <br/> org.opendaylight.controller.md.sal.common.api.data.TransactionChainClosedException: Cannot write into transaction.<br/> at org.opendaylight.openflowplugin.impl.device.TransactionChainManager.writeToTransaction(TransactionChainManager.java:233)<span class="error">&#91;183:org.opendaylight.openflowplugin.impl:0.3.0.SNAPSHOT&#93;</span><br/> at org.opendaylight.openflowplugin.impl.device.DeviceContextImpl.writeToTransaction(DeviceContextImpl.java:223)<span class="error">&#91;183:org.opendaylight.openflowplugin.impl:0.3.0.SNAPSHOT&#93;</span><br/> at org.opendaylight.openflowplugin.impl.device.DeviceContextImpl.processPortStatusMessage(DeviceContextImpl.java:331)<span class="error">&#91;183:org.opendaylight.openflowplugin.impl:0.3.0.SNAPSHOT&#93;</span><br/> at org.opendaylight.openflowplugin.impl.device.listener.OpenflowProtocolListenerFullImpl.onPortStatusMessage(OpenflowProtocolListenerFullImpl.java:94)<span class="error">&#91;183:org.opendaylight.openflowplugin.impl:0.3.0.SNAPSHOT&#93;</span><br/> at org.opendaylight.openflowjava.protocol.impl.core.connection.ConnectionAdapterImpl.consumeDeviceMessage(ConnectionAdapterImpl.java:129)<span class="error">&#91;173:org.opendaylight.openflowjava.openflow-protocol-impl:0.8.0.SNAPSHOT&#93;</span><br/> at org.opendaylight.openflowjava.protocol.impl.core.connection.AbstractConnectionAdapterStatistics.consume(AbstractConnectionAdapterStatistics.java:66)<span class="error">&#91;173:org.opendaylight.openflowjava.openflow-protocol-impl:0.8.0.SNAPSHOT&#93;</span><br/> at org.opendaylight.openflowjava.protocol.impl.core.connection.ConnectionAdapterImpl.consume(ConnectionAdapterImpl.java:43)<span class="error">&#91;173:org.opendaylight.openflowjava.openflow-protocol-impl:0.8.0.SNAPSHOT&#93;</span><br/> at org.opendaylight.openflowjava.protocol.impl.core.DelegatingInboundHandler.channelRead(DelegatingInboundHandler.java:46)<span class="error">&#91;173:org.opendaylight.openflowjava.openflow-protocol-impl:0.8.0.SNAPSHOT&#93;</span><br/> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)<span class="error">&#91;116:io.netty.transport:4.0.37.Final&#93;</span><br/> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:328)<span class="error">&#91;116:io.netty.transport:4.0.37.Final&#93;</span><br/> at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:321)<span class="error">&#91;116:io.netty.transport:4.0.37.Final&#93;</span><br/> at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)<span class="error">&#91;116:io.netty.transport:4.0.37.Final&#93;</span><br/> at org.opendaylight.openflowjava.protocol.impl.core.connection.AbstractOutboundQueueManager.channelRead(AbstractOutboundQueueManager.java:195)<span class="error">&#91;173:org.opendaylight.openflowjava.openflow-protocol-impl:0.8.0.SNAPSHOT&#93;</span><br/> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)<span class="error">&#91;116:io.netty.transport:4.0.37.Final&#93;</span><br/> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:328)<span class="error">&#91;116:io.netty.transport:4.0.37.Final&#93;</span><br/> at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:321)<span class="error">&#91;116:io.netty.transport:4.0.37.Final&#93;</span><br/> at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)<span class="error">&#91;117:io.netty.codec:4.0.37.Final&#93;</span><br/> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)<span class="error">&#91;116:io.netty.transport:4.0.37.Final&#93;</span><br/> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:328)<span class="error">&#91;116:io.netty.transport:4.0.37.Final&#93;</span><br/> at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:321)<span class="error">&#91;116:io.netty.transport:4.0.37.Final&#93;</span><br/> at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293)<span class="error">&#91;117:io.netty.codec:4.0.37.Final&#93;</span><br/> at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267)<span class="error">&#91;117:io.netty.codec:4.0.37.Final&#93;</span><br/> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)<span class="error">&#91;116:io.netty.transport:4.0.37.Final&#93;</span><br/> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:328)<span class="error">&#91;116:io.netty.transport:4.0.37.Final&#93;</span><br/> at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:321)<span class="error">&#91;116:io.netty.transport:4.0.37.Final&#93;</span><br/> at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293)<span class="error">&#91;117:io.netty.codec:4.0.37.Final&#93;</span><br/> at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267)<span class="error">&#91;117:io.netty.codec:4.0.37.Final&#93;</span><br/> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)<span class="error">&#91;116:io.netty.transport:4.0.37.Final&#93;</span><br/> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:328)<span class="error">&#91;116:io.netty.transport:4.0.37.Final&#93;</span><br/> at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:321)<span class="error">&#91;116:io.netty.transport:4.0.37.Final&#93;</span><br/> at io.netty.handler.timeout.ReadTimeoutHandler.channelRead(ReadTimeoutHandler.java:152)<span class="error">&#91;119:io.netty.handler:4.0.37.Final&#93;</span><br/> at org.opendaylight.openflowjava.protocol.impl.core.IdleHandler.channelRead(IdleHandler.java:39)<span class="error">&#91;173:org.opendaylight.openflowjava.openflow-protocol-impl:0.8.0.SNAPSHOT&#93;</span><br/> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)<span class="error">&#91;116:io.netty.transport:4.0.37.Final&#93;</span><br/> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:328)<span class="error">&#91;116:io.netty.transport:4.0.37.Final&#93;</span><br/> at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:321)<span class="error">&#91;116:io.netty.transport:4.0.37.Final&#93;</span><br/> at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1280)<span class="error">&#91;116:io.netty.transport:4.0.37.Final&#93;</span><br/> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)<span class="error">&#91;116:io.netty.transport:4.0.37.Final&#93;</span><br/> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:328)<span class="error">&#91;116:io.netty.transport:4.0.37.Final&#93;</span><br/> at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:890)<span class="error">&#91;116:io.netty.transport:4.0.37.Final&#93;</span><br/> at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:879)<span class="error">&#91;120:io.netty.transport-native-epoll:4.0.37.Final&#93;</span><br/> at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:387)<span class="error">&#91;120:io.netty.transport-native-epoll:4.0.37.Final&#93;</span><br/> at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:296)<span class="error">&#91;120:io.netty.transport-native-epoll:4.0.37.Final&#93;</span><br/> at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:112)<span class="error">&#91;115:io.netty.common:4.0.37.Final&#93;</span><br/> at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:145)<span class="error">&#91;115:io.netty.common:4.0.37.Final&#93;</span><br/> at java.lang.Thread.run(Thread.java:745)<span class="error">&#91;:1.8.0_101&#93;</span> </p> <p>The packets received at the ports were dropped as the status was reported as down.</p> <p>2016-08-28 19:35:13,879 | WARN | Runner: VTN Main | VBridge | 194 - org.opendaylight.vtn.manager.implementation - 0.5.0.SNAPSHOT | vBridge:Tenant1/vBridge1: Drop packet because egress port is down: src=0e:d0:3e:75:86:41, dst=52:5b:30:b8:d8:0b, port=openflow:2:1, type=0x806, vlan=0</p> <p>this is a blocker for VTN testing in OF 1.0 switches.</p> <p>Operating System: All<br/> Platform: All</p> OPNFLWPLUG-760

OVS Port remains DOWN in OpenFlow inventory when using Mininet 2.1.0 with OVS 2.0 OF1.0

Bug Resolved Cannot Reproduce Karthik Sivasamy Karthik Sivasamy Wed, 31 Aug 2016 13:29:44 +0000 Mon, 27 Sep 2021 09:01:53 +0000 Mon, 16 Oct 2017 16:26:31 +0000 General 0 7 <p>Have put in a fix to get more information on the port whose status is being missed:<br/> <a href="https://git.opendaylight.org/gerrit/#/c/45024/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/45024/</a></p> <p>Please try it out with this and share the logs for us to analyse the failure to process the port status message.</p> <p>However since :<br/> <a href="https://logs.opendaylight.org/releng/jenkins092/vtn-csit-1node-manager-only-boron/637/archives/karaf.log.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/vtn-csit-1node-manager-only-boron/637/archives/karaf.log.gz</a><br/> shows that there are 34 flow addition failures of which <br/> a. 19 are due to error type BADACTION code BADSETARGUMENT<br/> b. 10 due to Device disconnected<br/> c. 5 due to java.util.concurrent.CancellationException: Task was cancelled</p> <p>Could you guys also take a look and ascertain that<br/> 1. the moment flow is being pushed the switch is in connected state<br/> 2. flows that exist in the config DS are written to the switch if the switch connects back<br/> 3. The port status is reflected correctly - as in when it is up , it's actually up in the switch(OVS ) <br/> 4. The actions are supported in the switch (OVS) on which they are being pushed.</p> <p>I'm sharing the outcome of my investigation so far.</p> <p>I think the OpenFlow plugin failed to process a port status message because the message came from a switch before it started the service as MASTER for the switch.<br/> As a result, OpenFlow plugin failed to notify the VTN feature of a port creation, so the VTN feature didn't forward a packet as expected.<br/> I guess that a patch, which was merged recently, has changed the way to start the service as MASTER, and introduced this issue.</p> <ol> <li> <ol> <li>Detailed information</li> </ol> </li> </ol> <p>Here is detailed information of my investigation.</p> <p>I've checked the following log file on a failed job.</p> <p><a href="https://logs.opendaylight.org/sandbox/jenkins091/vtn-csit-3node-manager-all-boron/6/archives/odl1_karaf.log.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/sandbox/jenkins091/vtn-csit-3node-manager-all-boron/6/archives/odl1_karaf.log.gz</a></p> <p>In this job, the VTN feature wrongly judged a port "openflow:2:1" was down, because probably OFP didn't notify the VTN Manager of the port creation.</p> <p>I saw the following log messages on the log file.</p> <p>2016-09-01 18:16:24,099 | DEBUG | entLoopGroup-5-3 | TransactionChainManager | 183 - org.opendaylight.openflowplugin.impl - 0.3.0.SNAPSHOT | WriteTx is null for node KeyedInstanceIdentifier</p> {targetType=interface org.opendaylight.yang.gen.v1.urn.opendaylight.inventory.rev130819.nodes.Node, path=[org.opendaylight.yang.gen.v1.urn.opendaylight.inventory.rev130819.Nodes, org.opendaylight.yang.gen.v1.urn.opendaylight.inventory.rev130819.nodes.Node[key=NodeKey [_id=Uri [_value=openflow:2]]]]} <p>. Write data for KeyedInstanceIdentifier</p> {targetType=interface org.opendaylight.yang.gen.v1.urn.opendaylight.inventory.rev130819.node.NodeConnector, path=[org.opendaylight.yang.gen.v1.urn.opendaylight.inventory.rev130819.Nodes, org.opendaylight.yang.gen.v1.urn.opendaylight.inventory.rev130819.nodes.Node[key=NodeKey [_id=Uri [_value=openflow:2]]], org.opendaylight.yang.gen.v1.urn.opendaylight.inventory.rev130819.node.NodeConnector[key=NodeConnectorKey [_id=Uri [_value=openflow:2:1]]]]} <p> was not realized.<br/> 2016-09-01 18:16:24,099 | WARN | entLoopGroup-5-3 | DeviceContextImpl | 183 - org.opendaylight.openflowplugin.impl - 0.3.0.SNAPSHOT | Error processing port status message: <br/> org.opendaylight.controller.md.sal.common.api.data.TransactionChainClosedException: Cannot write into transaction.</p> <p>I think the OpenFlow plugin failed to process port status message, because it failed to get a WriteTransaction.<br/> I think the OpenFlow plugin failed to get the WriteTransaction, because it was before it started the service as MASTER for the siwtch "openflow:2".</p> <p>Actually, I saw the following log message 160 msec after the above log message.</p> <p>2016-09-01 18:16:24,260 | INFO | ult-dispatcher-2 | LifecycleServiceImpl | 183 - org.opendaylight.openflowplugin.impl - 0.3.0.SNAPSHOT | ========== Starting clustering MASTER services for node openflow:2 ==========</p> <ol> <li> <ol> <li>Related code</li> </ol> </li> </ol> <p> <span class="error">&#91;openflowplugin-impl/src/main/java/org/opendaylight/openflowplugin/impl/device/TransactionChainManager.java&#93;</span></p> <p> 223 &lt;T extends DataObject&gt; void writeToTransaction(final LogicalDatastoreType store,<br/> 224 final InstanceIdentifier&lt;T&gt; path,<br/> 225 final T data,<br/> 226 final boolean createParents){<br/> 227 final WriteTransaction writeTx = getTransactionSafely();<br/> 228 if (writeTx != null) {<br/> 229 LOG.trace("writeToTransaction called with path {} ", path);<br/> 230 writeTx.put(store, path, data, createParents);<br/> 231 } else {<br/> 232 LOG.debug("WriteTx is null for node {}. Write data for {} was not realized.", nodeII, path);<br/> 233 throw new TransactionChainClosedException("Cannot write into transaction.");<br/> 234 }<br/> 235 }</p> <p> 258 @Nullable<br/> 259 private WriteTransaction getTransactionSafely() {<br/> 260 synchronized (txLock) {<br/> 261 if (wTx == null &amp;&amp; TransactionChainManagerStatus.WORKING.equals(transactionChainManagerStatus)) {<br/> 262 if (wTx == null &amp;&amp; txChainFactory != null) </p> { 263 wTx = txChainFactory.newWriteOnlyTransaction(); 264 } <p> 265 }<br/> 266 }<br/> 267 return wTx;<br/> 268 }<br/> 269</p> <p>(In reply to Shuva Jyoti Kar from comment #1)<br/> &gt; However since :<br/> &gt; <a href="https://logs.opendaylight.org/releng/jenkins092/vtn-csit-1node-manager-only-" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/vtn-csit-1node-manager-only-</a><br/> &gt; boron/637/archives/karaf.log.gz<br/> &gt; shows that there are 34 flow addition failures of which <br/> &gt; a. 19 are due to error type BADACTION code BADSETARGUMENT<br/> &gt; b. 10 due to Device disconnected<br/> &gt; c. 5 due to java.util.concurrent.CancellationException: Task was cancelled</p> <p>Thank you for check the log messages!</p> <p>However, all those flow addition failures are not related to the CSIT test failures.<br/> So we should ignore all those flow addition failures in this bug report.</p> <p>In the VTN CSIT, firstly some tests are done with OF10 switches, and secondly some tests are done with OF13 switches.</p> <p>The test failures which this bug report is handling were occurred in the test with OF10.<br/> On the other hand, all flow addition failures which you pointed out were occurred in the test OF13, and those flow addition failures do not make the CSIT failed.</p> <p>And, the following two types of error messages are expected. In the test, test scripts stop and start mininet a few times to change its topology to test many scenarios. On an event to stop mininet, the following types of error messages are output, but that is expected, and that doesn't introduce any test failures.</p> <p>&gt; b. 10 due to Device disconnected<br/> &gt; c. 5 due to java.util.concurrent.CancellationException: Task was cancelled</p> <p>On the other hand, the following error is not good.<br/> VTN project needs to investigate this error, although this doesn't make the result of CSIT failed, so it's not a major bug probably. I'm guessing it's a bug of the test code of the CSIT. I'm filing the new bug report to deal with this issue.</p> <p>&gt; a. 19 are due to error type BADACTION code BADSETARGUMENT</p> <p>Anyway, as I explained in the previous comment in the #c2, I think the root cause of the test failure of CSIT is that the OpenFlow Plugin failed to process port status message, and failed to put correct port information into the operational DS. So I think we need to focus and fix this issue.</p> <p>Is there an ETA for this bug and someone assigned to fix?</p> <p>After more testing this bug only reproduces with OVS 2.0 in OF1.0 (default) mode the first time you start mininet after controller restarts. I have added test in OF suite to detect this: </p> <p><a href="https://git.opendaylight.org/gerrit/#/c/45193/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/45193/</a></p> <p>BR/Luis</p> <p>To better assess the impact of this bug and fix, could someone from your team please help us identify the following:<br/> Severity: Could you elaborate on the severity of this bug? Is this a BLOCKER such that we cannot release Boron without it? Is there a workaround such that we can write a release note and fix in future Boron SR1?<br/> Testing: Could you also elaborate on the testing of this patch? How extensively has this patch been tested? Is it covered by any unit tests or system tests? <br/> Impact: Does this fix impact any dependent projects?</p> <p>I just see 1 port status failing in Beryllium:</p> <p><a href="https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-1node-flow-services-only-beryllium/1827/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-1node-flow-services-only-beryllium/1827/</a></p> <p>This would mean:</p> <ul class="alternate" type="square"> <li>The issue may not be a regression.</li> <li>It is probably some missbehavior in OVS 2.0 OF1.0</li> </ul> <p>Therefore I downgrade the importance from Blocker to Major.</p> <p>Has this bug been verified as fixed in the latest Boron RC 3.1 Build?</p> <p>Mininet 2.1.0 with OVS 2.0 OF1.0 has this strange behavior that switch s2 connects to controller with port s2-eth1 down. After some time the switch sends a port status update to controller who seems to miss to update the OpenFlow inventory. I still believe there is something odd in the controller with this port status update but since this only happens with very specific (and old) version of mininet and OVS we can definitely lower the priority of this.</p> <p>Can someone retest this again on latest master? Patch that was solving issues with missed/early port status messages was merged and it was solving similar issues as ones mentioned in this bug.</p> <p>Closing this as there was no update.</p> Development External issue ID 6595 External issue URL Rank 0|i03353: