Operating System: All
Platform: All

Hi Luis,

Is this for He plugin or Li plugin or both?

Both, see the mail I just sent.

Luis , is this issue observed with OF-HA ?

Also Luis, is this issue sporadic or consistent ?

If you see above, switch connects to the 3 nodes.

It is consistently happening with Li plugin and lately very rare in He plugin.

Hi Luis,

Following are the procedure I have followed for the bug:
• Got the latest beryllium distribution.
• Installed openflowplugin features.
• Added bridge and set controllers to the nodes. [[Control node and two compute nodes]]
[stack@compute2 devstack]$ sudo ovs-vsctl --no-wait add-br br-int
[stack@compute2 devstack]$ sudo ovs-vsctl set bridge br-int protocols=["OpenFlow13"]
[stack@compute2 devstack]$ sudo ovs-vsctl set-controller br-int tcp:10.106.138.154:6653 tcp:10.106.138.155:6653 tcp:10.106.138.110:6653
• Isolate owner from other nodes using iptables.
sudo iptables -A OUTPUT -d 10.106.138.110 -j DROP
sudo iptables -A OUTPUT -d 10.106.138.154 -j DROP
The members are removed from the entity.
• Rejoin the controllers to the cluster.
sudo iptables -D OUTPUT -d 10.106.138.110 -j DROP
sudo iptables -D OUTPUT -d 10.106.138.154 -j DROP
The members are getting added to the entity as candidates.
PFA for the entity owner observation.

Since you has mentioned, this bug is sporadically reproduced, I repeated the tests for 3 times each. (Li and He Plugin).
The members are isolated and rejoined properly from/to the entity list.

Queries:
1. You have mentioned “Either 4) or 6) fails”. Is that fails means, the members are not isolated or rejoined properly from/to the entity list?
2. Should I look at CSIT regarding this bug?

Thanks,
Arthi

Attachment Bug#6177_Observation.txt has been added with description: Entity Owner Observation Document

OK, lets track the yellow builds on this job (the red are infra issues):

https://jenkins.opendaylight.org/releng/job/openflowplugin-csit-3node-clustering-only-beryllium/

As I commented before, the issue seems to happen very much with the Boron (Li redesign) plugin but not often with Beryllium (He redesign).

Hi guys, this should be solved within OPNFLWPLUG-659.
(using cluster singleton service instead of EOS)

We have tested https://git.opendaylight.org/gerrit/#/c/44664/ and there are still issues. Jozef will continue working on that. It is not needed to merge it yet.

Thanks,
Miro

The issue of isolate + rejoin is present in Beryllium -> It is not regression in Boron -> It is not blocker. Still good to fix if we can.

Marked as dependent on MDSAL-195 which is probably the real cause.

After fixing the test, the latest and probably the only issue observed is a change in the owner when an instance is rejoined to the cluster, like for example if a rejoin member 1 that was previously isolated, the device owner changes from 2 to 3. Kind of weird but this is what I see in the test.

Setup:
1 control and 2 compute nodes

Distro: distribution-karaf-0.4.4-20160916.220725-251.tar.gz

I've tested the scenario and observed the ownership is not been changed.
Say,
"candidate":[

,

,

],"owner":"member-1"}

I'm isolating the member-1
sudo iptables -A OUTPUT -d 10.106.138.136 -j DROP

Then member-2 becomes owner.

And while rejoining member-1
sudo iptables -D OUTPUT -d 10.106.138.136 -j DROP

The ownership is not been changed, It's still member-2
I've tested for 3 times. I observed the same result.

Attachment 6177.txt has been added with description: 6177_observation

So it is been a while I do not look at this issue and I think you are right, I do not see the change of owner anymore but I still see some sporadic failures of isolated member not joining the cluster when iptables rule is deleted:

https://logs.opendaylight.org/releng/jenkins092/openflowplugin-csit-3node-clustering-only-boron/724/archives/log.html.gz

Manual testing and carbon/boron jenkin's jobs look ok.