https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/OPNFLWPLUG-891 OpenFlowPlugin <p>When defining large number of networks/subnets in bulk, OVS connection flaps frequently and eventually remains disconnected.</p> <p>Environment: single openstack node and single ODL.<br/> ODL version: carbon - <br/> distribution-karaf-0.6.0-20170516.135813-5306.tar.gz</p> <p>To reproduce:</p> <p>1) using script to define large number of networks/subnets:</p> <p>neutron net-create vx-net$j-$i --provider:network_type vxlan<br/> neutron subnet-create vx-net$j-$i 10.$j.$i.0/24 -<del>name vx-subnet$j</del>$i --dns-nameserver 8.8.8.8</p> <p>where j is <span class="error">&#91;1..15&#93;</span> and j is <span class="error">&#91;1..100&#93;</span></p> <p>2) Around 1000 networks, OVS connection flaps and eventually remains disconneted.</p> <p>Operating System: All<br/> Platform: All</p> OPNFLWPLUG-891

Xid cannot be reserved after OVS connection flaps

Bug Resolved Done Jozef Bacigal Vinh Nguyen Tue, 16 May 2017 22:37:19 +0000 Mon, 27 Sep 2021 09:02:02 +0000 Mon, 28 Aug 2017 12:13:08 +0000 General 0 11 <p>Attachment karaf.log.tar has been added with description: karar.log.tar</p> <p>Hi Vinh, </p> <p>can you please run this again on latest snapshots with bigger log size and DEBUG log for org.opendayligh.openflowplugin.impl</p> <p>Thanks</p> <p>Jozef</p> <p>Hi Jozef, I am not able to reproduce this problem with the latest snapshot. However since this issue happened intermittently when I logged it, I'll try some more runs to make sure it is gone in the latest code base</p> <p>Attachment karaf.log.18.7z has been added with description: karaf.log.18</p> <p>Attachment karaf.log.17.7z has been added with description: karaf.log.17</p> <p>Attachment karaf.log.16.7z has been added with description: karaf.log.16</p> <p>Attachment karaf.log.15.7z has been added with description: karaf.log.15</p> <p>Attachment karaf.log.14.7z has been added with description: karaf.log.14</p> <p>Attachment karaf.log.13.7z has been added with description: karaf.log.13</p> <p>Attachment karaf.log.12.7z has been added with description: karaf.log.12</p> <p>Attachment karaf.log.11.7z has been added with description: karaf.log.11</p> <p>Attachment karaf.log.10.7z has been added with description: karaf.log.10</p> <p>I cannot reproduce the original issue with latest ODL snapshot. However I encounter another issue:</p> <p>When defining large number of networks/subnets in bulk, OVS connection flaps frequently and new requests for network/subnet creation results in ERROR/WARNING in log files:</p> <p>"No queue present, failing request"<br/> "Xid cannot be reserved for new RequestContext" </p> <p>Environment: single openstack node and single ODL.<br/> ODL version: distribution-karaf-0.6.1-20170609.000930-229.tar.gz</p> <p>To reproduce:</p> <p>1) using script to define large number of networks/subnets:</p> <p>neutron net-create vx-net$j-$i --provider:network_type vxlan<br/> neutron subnet-create vx-net$j-$i 10.$j.$i.0/24 -<del>name vx-subnet$j</del>$i --dns-nameserver 8.8.8.8</p> <p>where j is <span class="error">&#91;1..30&#93;</span> and j is <span class="error">&#91;1..100&#93;</span></p> <p>2) OVS connection to OFPlugin flaps several times but ODL recovers and keeps up the with continuous network creation requests</p> <p>3) Around 3000 network creations, OVS connection flaps, this time new network creation requests result in ERROR/WARNING in karaf.log:</p> <p>"No queue present, failing request"<br/> "Xid cannot be reserved for new RequestContext" </p> <p>I attached the karaf log files captured before and when the issus happens.</p> <p>Some interesting notes:</p> <p>1) The "Xid cannot be reserved " issue happens in karaf.log.10</p> <p>2) There are two consecutive ConnectionEvent for "Device connected" in the last connection flap.</p> <p>Normally, the karaf log shows two events: one for disconnection and one for connection when the connection flaps:</p> <p>&gt; grep ConnectionEvent <br/> karaf.log.18:2017-06-13 03:26:05,245 | INFO | entLoopGroup-7-4 | SystemNotificationsListenerImpl | 306 - org.opendaylight.openflowplugin.impl - 0.4.1.SNAPSHOT | ConnectionEvent: Connection closed by device, Device:/10.138.0.2:49318, NodeId:openflow:158835981449481<br/> karaf.log.18:2017-06-13 03:26:05,868 | INFO | entLoopGroup-7-5 | DeviceManagerImpl | 306 - org.opendaylight.openflowplugin.impl - 0.4.1.SNAPSHOT | ConnectionEvent: Device connected to controller, Device:/10.138.0.2:53148, NodeId:Uri <span class="error">&#91;_value=openflow:158835981449481&#93;</span><br/> karaf.log.15:2017-06-13 04:54:15,286 | INFO | entLoopGroup-7-5 | SystemNotificationsListenerImpl | 306 - org.opendaylight.openflowplugin.impl - 0.4.1.SNAPSHOT | ConnectionEvent: Connection closed by device, Device:/10.138.0.2:53148, NodeId:openflow:158835981449481<br/> karaf.log.15:2017-06-13 04:54:15,827 | INFO | entLoopGroup-7-6 | DeviceManagerImpl | 306 - org.opendaylight.openflowplugin.impl - 0.4.1.SNAPSHOT | ConnectionEvent: Device connected to controller, Device:/10.138.0.2:60476, NodeId:Uri <span class="error">&#91;_value=openflow:158835981449481&#93;</span><br/> karaf.log.10:2017-06-13 06:10:25,663 | INFO | entLoopGroup-7-8 | DeviceManagerImpl | 306 - org.opendaylight.openflowplugin.impl - 0.4.1.SNAPSHOT | ConnectionEvent: Device connected to controller, Device:/10.138.0.2:36790, NodeId:Uri <span class="error">&#91;_value=openflow:158835981449481&#93;</span></p> <p>3) In between the two "Device Connected" ConnectionEvents described above, there is an handshake attempt from the OVS. OF Plugin sees thatthis device is already connected and closing all connections:</p> <p>2017-06-13 06:10:23,680 | INFO | entLoopGroup-7-7 | ConnectionAdapterImpl | 295 - org.opendaylight.openflowjava.openflow-protocol-impl - 0.9.1.SNAPSHOT | Hello received<br/> 2017-06-13 06:10:23,681 | DEBUG | entLoopGroup-7-7 | nflowProtocolListenerInitialImpl | 306 - org.opendaylight.openflowplugin.impl - 0.4.1.SNAPSHOT | processing HELLO.xid: 36 from device /10.138.0.2:36788<br/> 2017-06-13 06:10:23,682 | DEBUG | entLoopGroup-7-7 | HandshakeListenerImpl | 306 - org.opendaylight.openflowplugin.impl - 0.4.1.SNAPSHOT | handshake succeeded: /10.138.0.2:36788<br/> 2017-06-13 06:10:23,683 | DEBUG | entLoopGroup-7-7 | HandshakeListenerImpl | 306 - org.opendaylight.openflowplugin.impl - 0.4.1.SNAPSHOT | succeeded by getting sweep barrier after post-handshake for device openflow:158835981449481<br/> 2017-06-13 06:10:23,683 | INFO | entLoopGroup-7-7 | ContextChainHolderImpl | 306 - org.opendaylight.openflowplugin.impl - 0.4.1.SNAPSHOT | Device openflow:158835981449481 connected.<br/> 2017-06-13 06:10:23,683 | WARN | entLoopGroup-7-7 | ContextChainHolderImpl | 306 - org.opendaylight.openflowplugin.impl - 0.4.1.SNAPSHOT | Device openflow:158835981449481 already connected. Closing all connection to the device.</p> <p>Recheck this please after this patch</p> <p><a href="https://git.opendaylight.org/gerrit/#/c/57814/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/57814/</a></p> <p>The problem still exists after the patch:</p> <p><a href="https://git.opendaylight.org/gerrit/#/c/57814/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/57814/</a></p> <p>The "Xid cannot be reserved " problem still happens after the duplicate connection is received:</p> <p>"Device openflow:158835981449481 already connected. Closing all connection to the device"</p> <p>OK and now the double connection problem <img class="emoticon" src="https://jira.opendaylight.org/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p><a href="https://git.opendaylight.org/gerrit/#/c/59557/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/59557/</a></p> <p>Try please this patch.</p> <p>Thanks</p> <p>Karaf.log for patch 59557</p> <p>Attachment karaf.log.7z has been added with description: karaf.log.patch59557</p> <p>The patch 59557 doesn't fix the problem. New flows are still not pushed when new connection handshake received for already-connected OVS device.</p> <p>Please see the attached karaf logs (last 10 logs, the already-connected connection issue happens in karaf.log file)</p> <p>(In reply to Vinh Nguyen from comment #16)<br/> &gt; Created attachment 1896 <span class="error">&#91;details&#93;</span><br/> &gt; karaf.log.patch59557<br/> &gt; <br/> &gt; Karaf.log for patch 59557</p> <p>Please ignore this attachment</p> <p>Attachment karaf.7z has been added with description: New karaf.log for 59557</p> <p>(In reply to Vinh Nguyen from comment #17)<br/> &gt; The patch 59557 doesn't fix the problem. New flows are still not pushed when<br/> &gt; new connection handshake received for already-connected OVS device.<br/> &gt; <br/> &gt; Please see the attached karaf logs (last 10 logs, the already-connected<br/> &gt; connection issue happens in karaf.log file)</p> <p>The error messages "Xid cannot be reserved.." are not seen in the logs but the problem now seems that odl connection are dropped but new connection is not processed properly.</p> <p>Closing this, connection bug is 8805</p> Development External issue ID 8478 External issue URL Issue Type ODL SR Target Milestone Rank 0|i033y7: