https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/OPNFLWPLUG-790 OpenFlowPlugin <p>openflowplugin fails to update port status on creation of VM using netvirt.<br/> it always happens on the same compute host. VMs created on other computes work fine.</p> <p>this is the relevant lines in the jaraf.log<br/> 2016-10-09 07:07:17,263 | WARN | ntLoopGroup-9-15 | DeviceContextImpl | 265 - org.opendaylight.openflowplugin.impl - 0.3.0.SNAPSHOT | Error processing port status message for port 13 on device Uri <span class="error">&#91;_value=openflow:79818282140722&#93;</span> : java.lang.IllegalStateException: Previous transaction member-1-datastore-operational-fe-0-chn-591-txn-6 is not ready yet<br/> 2016-10-09 07:07:17,364 | WARN | Thread-16795 | DeviceContextImpl | 265 - org.opendaylight.openflowplugin.impl - 0.3.0.SNAPSHOT | Error processing port status message for 0 on port 13 on device Uri <span class="error">&#91;_value=openflow:79818282140722&#93;</span> : java.lang.IllegalStateException: Previous transaction member-1-datastore-operational-fe-0-chn-591-txn-6 is not ready yet<br/> 2016-10-09 07:07:17,475 | WARN | Thread-16795 | DeviceContextImpl | 265 - org.opendaylight.openflowplugin.impl - 0.3.0.SNAPSHOT | Error processing port status message for 1 on port 13 on device Uri <span class="error">&#91;_value=openflow:79818282140722&#93;</span> : java.lang.IllegalStateException: Previous transaction member-1-datastore-operational-fe-0-chn-591-txn-6 is not ready yet<br/> 2016-10-09 07:07:17,576 | WARN | Thread-16795 | DeviceContextImpl | 265 - org.opendaylight.openflowplugin.impl - 0.3.0.SNAPSHOT | Error processing port status message for 2 on port 13 on device Uri <span class="error">&#91;_value=openflow:79818282140722&#93;</span> : java.lang.IllegalStateException: Previous transaction member-1-datastore-operational-fe-0-chn-591-txn-6 is not ready yet<br/> 2016-10-09 07:07:17,576 | WARN | Thread-16795 | DeviceContextImpl | 265 - org.opendaylight.openflowplugin.impl - 0.3.0.SNAPSHOT | Failed to update port status for port 13 on device Uri <span class="error">&#91;_value=openflow:79818282140722&#93;</span> even after 3 retries</p> <p>Operating System: All<br/> Platform: All</p> OPNFLWPLUG-790

failed to update port status - all new OVS ports fail from this point

Bug Resolved Done Arnon Yahel Arnon Yahel Sun, 9 Oct 2016 13:19:25 +0000 Mon, 27 Sep 2021 09:01:55 +0000 Mon, 19 Jun 2017 12:19:07 +0000 General 0 10 <p>Attachment karaf.logs.tar has been added with description: karaf logs</p> <p>I also encountered that problem:<br/> repetetive creating of VMs failed</p> <p>Attachment qsvc.zip has been added with description: q-svc log (fail @08/11, 9:22)</p> <p>Attachment karaf.zip has been added with description: Karaf log (fail @08/11, 9:22)</p> <p>This bug is reproduced when using this script:<br/> for i in </p> {1..1000} <p>;do<br/> date;<br/> neutron net-create net1;<br/> neutron subnet-create net1 192.168.1.0/24 --name subnet1;<br/> nova boot --image cirros-0.3.4-x86_64-uec --flavor m1.nano --nic net-name=net1 VM$i;<br/> IP_NETNS=`ip netns | head -n 1`;<br/> sleep 15;<br/> echo sudo ip netns exec ${IP_NETNS} ping -c 3 192.168.1.3;<br/> sudo ip netns exec ${IP_NETNS} ping -c 3 192.168.1.3;<br/> nova delete VM$i;<br/> neutron net-delete net1;<br/> done<br/> At some point, the update port failure appears and from that point, each new VM is not pingable.</p> <p>Attachment karaf.zip has been added with description: karaf.log - TransactionChainManager in DEBUG level</p> <p>openflowplugin can't figure it out - need to try to recreate with a running tcpdump and debug logs for whole openflowplugin.impl<br/> Shuva, I think this happened when using only 2 computes - will ask Dotan to try to recreate, have you tried his script though?</p> <p>From Shuva:<br/> <a href="https://lists.opendaylight.org/pipermail/openflowplugin-dev/2016-December/006436.html" class="external-link" target="_blank" rel="nofollow noopener">https://lists.opendaylight.org/pipermail/openflowplugin-dev/2016-December/006436.html</a></p> <p>"Sorry for the late response, got pulled into multiple things.<br/> The actual issue was reported as a part of <a href="https://jira.opendaylight.org/browse/OPNFLWPLUG-760" title="OVS Port remains DOWN in OpenFlow inventory when using Mininet 2.1.0 with OVS 2.0 OF1.0" class="issue-link" data-issue-key="OPNFLWPLUG-760"><del>OPNFLWPLUG-760</del></a>(<a href="https://bugs.opendaylight.org/show_bug.cgi?id=6595" class="external-link" target="_blank" rel="nofollow noopener">https://bugs.opendaylight.org/show_bug.cgi?id=6595</a>) wherein the OpenFlow plugin failed to process a port status message because the message came from a switch before it started the service as MASTER for the switch.</p> <p>Analysing the logs attached to this bug I was not able to come to any conclusion. To analyse the issue further we need the logs by setting the log level as<br/> log:set DEBUG org.opendaylight.openflowplugin.impl</p> <p>I assume that we see the issue when the number of switches(DPNs) are high, hence we need to check what is the typical delay between the MASTERSHIP grant and the ports coming up and adjust the delay accordingly.</p> <p>We can also get the relevant information in a wireshark capture that would demarcate when the ROLE_REQ is processed by the switch and when the PORT STATUS is sent."</p> <p>This was reproduced using Dotan's instructions, with only 1 compute (one devstack machine that is "all in one" - controller+compute).</p> <p>An updated karaf.log with all openflowplugin set to DEBUG was provided via mail on 12/8 (too large to attach here), as requested by Shuva.</p> <p>Updating to blocker to at least get openflowplugin's feedback as this bug has blocked us from proceeding several times. Thanks.</p> <p>Is this only for Carbon or does it apply to Boron-SR2 as well?</p> <p>(In reply to A H from comment #10)<br/> &gt; Is this only for Carbon or does it apply to Boron-SR2 as well?</p> <p>This applies to Boron SR2.</p> <p>When it occurs, that DPN is stuck in a zombie state and cannot be used further.<br/> Waiting for analysis/feedback from openflowplugin, details and reproduction info provided to Shuva below and by Koby, at least to figure out a workaround for SR2.</p> <p>As per the discussion on the following email thread below, this bug is being retargeted for Boron SR3:</p> <p><a href="https://lists.opendaylight.org/pipermail/openflowplugin-dev/2016-December/006490.html" class="external-link" target="_blank" rel="nofollow noopener">https://lists.opendaylight.org/pipermail/openflowplugin-dev/2016-December/006490.html</a></p> <p>Is this still blocking?</p> <p>I think we can say this is no longer blocking (at least for netvirt), since we have a workaround (disabling statistics polling in openflowplugin). I believe it is still critical, as statistics collection is on by default, and users may encounter those issues which make the entire OVS unusable.</p> <p>CC'd Sam &amp; Alon - maybe they'd like to comment as well.</p> <p>Changing importance to critical from blocking as Koby said it is non blocking anymore.</p> <p>Is this on boron or just master?</p> <p>Is the workaround a simple config option?</p> <p>What deployment and setup does this bug happen?</p> <p>Answers to the above would help to decide lowering the priority.</p> <p>Can you test this again on this patch please? <a href="https://git.opendaylight.org/gerrit/#/c/53905/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/53905/</a></p> <p>I added collecting of early port status updates and after initial statistics gathering is done these collected port status updates will be written to operational datastore.</p> <p>I will test this patch.<br/> usually, it takes about 1 day to reproduce.</p> <p>I'm sorry for the delay-<br/> I encountered several problems on testing it.<br/> As for now, it looks good.<br/> I will give another update when finish testing it</p> <p>Okay great</p> <p>It looks good.<br/> The message of "failed to update port status" was not reproduced<br/> Thanks</p> <p>So patch was merged, is it working and can this one be closed?</p> <p>Last patch will probably not work properly with statistics gathering disabled, so I need to make another one that tries to solve this better.</p> <p>GERRIT: <a href="https://git.opendaylight.org/gerrit/#/c/55868/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/55868/</a></p> <p>Patch was merged, but there is also cherry-pick for stable/carbon, so it needs to be updated and merged too before this can be closed: <a href="https://git.opendaylight.org/gerrit/#/c/56132/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/56132/</a></p> <p>Closing this.</p> <p>Forgot that this is still waiting for carbon merge, so leaving this in review.</p> Duplicate NETVIRT-260 UNIMGR-27 Development External issue ID 6908 External issue URL ODL SR Target Milestone Rank 0|i033br: