Operating System: All
Platform: All

Attachment karaf.logs.tar has been added with description: karaf logs

I also encountered that problem:
repetetive creating of VMs failed

Attachment qsvc.zip has been added with description: q-svc log (fail @08/11, 9:22)

Attachment karaf.zip has been added with description: Karaf log (fail @08/11, 9:22)

This bug is reproduced when using this script:
for i in

;do
date;
neutron net-create net1;
neutron subnet-create net1 192.168.1.0/24 --name subnet1;
nova boot --image cirros-0.3.4-x86_64-uec --flavor m1.nano --nic net-name=net1 VM$i;
IP_NETNS=`ip netns | head -n 1`;
sleep 15;
echo sudo ip netns exec ${IP_NETNS} ping -c 3 192.168.1.3;
sudo ip netns exec ${IP_NETNS} ping -c 3 192.168.1.3;
nova delete VM$i;
neutron net-delete net1;
done
At some point, the update port failure appears and from that point, each new VM is not pingable.

Attachment karaf.zip has been added with description: karaf.log - TransactionChainManager in DEBUG level

openflowplugin can't figure it out - need to try to recreate with a running tcpdump and debug logs for whole openflowplugin.impl
Shuva, I think this happened when using only 2 computes - will ask Dotan to try to recreate, have you tried his script though?

From Shuva:
https://lists.opendaylight.org/pipermail/openflowplugin-dev/2016-December/006436.html

"Sorry for the late response, got pulled into multiple things.
The actual issue was reported as a part of OPNFLWPLUG-760(https://bugs.opendaylight.org/show_bug.cgi?id=6595) wherein the OpenFlow plugin failed to process a port status message because the message came from a switch before it started the service as MASTER for the switch.

Analysing the logs attached to this bug I was not able to come to any conclusion. To analyse the issue further we need the logs by setting the log level as
log:set DEBUG org.opendaylight.openflowplugin.impl

I assume that we see the issue when the number of switches(DPNs) are high, hence we need to check what is the typical delay between the MASTERSHIP grant and the ports coming up and adjust the delay accordingly.

We can also get the relevant information in a wireshark capture that would demarcate when the ROLE_REQ is processed by the switch and when the PORT STATUS is sent."

This was reproduced using Dotan's instructions, with only 1 compute (one devstack machine that is "all in one" - controller+compute).

An updated karaf.log with all openflowplugin set to DEBUG was provided via mail on 12/8 (too large to attach here), as requested by Shuva.

Updating to blocker to at least get openflowplugin's feedback as this bug has blocked us from proceeding several times. Thanks.

Is this only for Carbon or does it apply to Boron-SR2 as well?

(In reply to A H from comment #10)
> Is this only for Carbon or does it apply to Boron-SR2 as well?

This applies to Boron SR2.

When it occurs, that DPN is stuck in a zombie state and cannot be used further.
Waiting for analysis/feedback from openflowplugin, details and reproduction info provided to Shuva below and by Koby, at least to figure out a workaround for SR2.

As per the discussion on the following email thread below, this bug is being retargeted for Boron SR3:

https://lists.opendaylight.org/pipermail/openflowplugin-dev/2016-December/006490.html

Is this still blocking?

I think we can say this is no longer blocking (at least for netvirt), since we have a workaround (disabling statistics polling in openflowplugin). I believe it is still critical, as statistics collection is on by default, and users may encounter those issues which make the entire OVS unusable.

CC'd Sam & Alon - maybe they'd like to comment as well.

Changing importance to critical from blocking as Koby said it is non blocking anymore.

Is this on boron or just master?

Is the workaround a simple config option?

What deployment and setup does this bug happen?

Answers to the above would help to decide lowering the priority.

Can you test this again on this patch please? https://git.opendaylight.org/gerrit/#/c/53905/

I added collecting of early port status updates and after initial statistics gathering is done these collected port status updates will be written to operational datastore.

I will test this patch.
usually, it takes about 1 day to reproduce.

I'm sorry for the delay-
I encountered several problems on testing it.
As for now, it looks good.
I will give another update when finish testing it

Okay great

It looks good.
The message of "failed to update port status" was not reproduced
Thanks

So patch was merged, is it working and can this one be closed?

Last patch will probably not work properly with statistics gathering disabled, so I need to make another one that tries to solve this better.

GERRIT: https://git.opendaylight.org/gerrit/#/c/55868/

Patch was merged, but there is also cherry-pick for stable/carbon, so it needs to be updated and merged too before this can be closed: https://git.opendaylight.org/gerrit/#/c/56132/

Closing this.

Forgot that this is still waiting for carbon merge, so leaving this in review.