Switch disconnect while mastership is being negotiated produces stale switch entry

This only happens in Carbon and it is tracked in the reconciliation suite:

https://logs.opendaylight.org/releng/jenkins092/openflowplugin-csit-3node-clustering-only-carbon/614/archives/log.html.gz#s1-s7

To reproduce:

1) Start 3 nodes cluster

2) Start 1 switch (s1) and connect it to 2 nodes (.101 and .102):

sudo ovs-vsctl set-controller s1 "tcp:192.168.0.101:6633" "tcp:192.168.0.102:6633"

3) Block communication owner to switch (in this example .101):

sudo iptables -A INPUT --source 192.168.0.101 -j DROP

4) Just after block communication remaining node to switch:

sudo iptables -A INPUT --source 192.168.0.102 -j DROP

5) Now observe the following after few seconds:

• The switch closes owner connection (.101).
• New mastership negotiation starts in remaining node (.102).
• The switch closes remaining connection (.102).

The result of the above is an stale switch in operational, only workaround is reboot entire cluster.

BR/Luis