Operating System: All
Platform: All

Attachment karaf_log_device_connection_bounce.txt has been added with description: Karaf log

to reproduce the above:

1) start mininet 1 switch (s1) pointing to any of the cluster instances (e.g. 192.168.0.101)
2) Run the following script in the mininet system to bounce connection quickly:

#!/bin/bash
sudo ovs-vsctl del-controller s1
sleep 0.2
sudo ovs-vsctl set-controller s1 "tcp:192.168.0.101"

Posted patch that will hopefully fix this issue: https://git.opendaylight.org/gerrit/#/c/45526 . Luis, can you recheck it?

So I partially fixed this issue with patch I posted earlier, but I think there is problem with Singleton (sometimes, it is sending SLAVE instead of MASTER) so I raised bug in mdsal here: https://bugs.opendaylight.org/show_bug.cgi?id=6710 and added it as blocker for this issue.

I see some NPEs:

2016-09-08 23:35:05,409 | ERROR | pool-31-thread-1 | ExecutionList | 65 - com.google.guava - 18.0.0 | RuntimeException while executing runnable com.google.common.util.concurrent.Futures$6@66487d12 with executor INSTANCE
java.lang.NullPointerException
at org.opendaylight.openflowplugin.impl.device.DeviceContextImpl.shutdownConnection(DeviceContextImpl.java:568)

but looks like that is fixed by Tomas's patch.

The attached log is from member-1 which wasn't the EOS shard leader - member-2 was. When device connections were dropped, I see the candidates for the ServiceEntityType removed for member-3 and member-2. I don't see the candidate removed for member-1 - I assume that's not expected? Maybe the NPE prevented it? In any event, it seems member-1 remained as candidate and thus the owner. As far as the EOS is concerned, this is correct.

I don't see any evidence of an entity with an owner that isn't a candidate as we saw in the CI test a couple weeks ago. However I did find an issue that can result in that scenario that I fixed with https://git.opendaylight.org/gerrit/#/c/45516/.

Added another patch on Gerrit: https://git.opendaylight.org/gerrit/#/c/46321
that will hopefully make unregistering process of cluster singleton services better and more stable, and that means that connection bounce will also work better.

Gerrit: https://git.opendaylight.org/gerrit/#/c/46390/

This should fix all errors with fast connection and disconnection of device. This patch depends on this https://git.opendaylight.org/gerrit/#/c/45638/ patch in controller and this patch https://git.opendaylight.org/gerrit/#/c/46175/ in mdsal. After these 3 will be merged, I think everything should be fine.

Thanks Tomas for spending time in this

So, both mdsal (https://git.opendaylight.org/gerrit/#/c/46175/) and controller (https://git.opendaylight.org/gerrit/#/c/45638/) changes what was required to solve this issue (and other related) issues was merged in master. So if you can Luis, can you test this connection bounce on my patch (https://git.opendaylight.org/gerrit/#/c/46390/) if it is really working?

I am testing your patch today, I will let you know the results.

(In reply to Luis Gomez from comment #9)
> I am testing your patch today, I will let you know the results.

Luis , any updates on this ?

This is fixed now.