Leader state changed events received in reversed order

When the ODL which is the leader of some shard is shutdown, one of the remaining ODL first receives two leader changed events, first event indicating new leader is selected and then the second event indicating the shard leader is null. The correct order of events received should be reversed.

The scenario is described below:

1) 6-nodes cluster (I am not sure the fact that cluster has more than 3 nodes is causing the problem because we always test with 3-nodes cluster)
2) ODL node 1 is leader of some shard, say ‘shard-default-config’
3) Now shutdown ODL1, the leader for ‘shard-default-config’ is transferred to ODL2
4) There are two events received by each node, first one indicates leader is null (current leader is no longer valid), the second for the new leader (ODL2) is selected.
5) Four of the remaining ODL nodes receive the ‘onLeaderStateChanged’ in that order
6) Problem: ODL3 node receives the ‘onLeaderStateChanged’ in reversed order. First it receives the event for new leader,
and then it receives the event for leader == null. This ODL node, based on the last event (leader==null), thinks that the ‘shard-default-config’ shard has no leader and throws exception when there is a read/write to the shard.

ReadFailedException

{message=Error executeRead ReadData for path /(urn:opendaylight:neutron?revision=2015-07-12)neutron/networks, errorList=[RpcError [message=Error executeRead ReadData for path /(urn:opendaylight:neutron?revision=2015-07-12)neutron/networks, severity=ERROR, errorType=APPLICATION, tag=operation-failed, applicationTag=null, info=null, cause=org.opendaylight.controller.md.sal.common.api.data.DataStoreUnavailableException: Shard member-3-shard-default-config currently has no leader. Try again later.]]}

Below is the excerpt from karaf log filtered on the ‘onLeaderStateChanged’ events when ODL1 is shut down. Please note the events for ‘shard-default-config’:

2016-11-14 04:32:35,263 | INFO | ult-dispatcher-6 | ShardManager | 195 - org.opendaylight.controller.sal-distributed-datastore - 1.4.1.Boron-SR1 | shard-manager-operational: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-3-shard-default-operational, leaderId=null, leaderPayloadVersion=5]
2016-11-14 04:32:35,263 | INFO | ult-dispatcher-6 | ShardManager | 195 - org.opendaylight.controller.sal-distributed-datastore - 1.4.1.Boron-SR1 | shard-manager-operational: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-3-shard-entity-ownership-operational, leaderId=null, leaderPayloadVersion=5]
2016-11-14 04:32:35,263 | INFO | ult-dispatcher-6 | ShardManager | 195 - org.opendaylight.controller.sal-distributed-datastore - 1.4.1.Boron-SR1 | shard-manager-operational: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-3-shard-toaster-operational, leaderId=null, leaderPayloadVersion=5]
2016-11-14 04:32:35,263 | INFO | ult-dispatcher-6 | ShardManager | 195 - org.opendaylight.controller.sal-distributed-datastore - 1.4.1.Boron-SR1 | shard-manager-operational: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-3-shard-inventory-operational, leaderId=null, leaderPayloadVersion=5]
2016-11-14 04:32:35,263 | INFO | ult-dispatcher-6 | ShardManager | 195 - org.opendaylight.controller.sal-distributed-datastore - 1.4.1.Boron-SR1 | shard-manager-operational: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-3-shard-topology-operational, leaderId=null, leaderPayloadVersion=5]
2016-11-14 04:32:35,318 | INFO | ult-dispatcher-6 | ShardManager | 195 - org.opendaylight.controller.sal-distributed-datastore - 1.4.1.Boron-SR1 | shard-manager-operational: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-3-shard-inventory-operational, leaderId=member-4-shard-inventory-operational, leaderPayloadVersion=5]
2016-11-14 04:32:35,318 | INFO | ult-dispatcher-6 | ShardManager | 195 - org.opendaylight.controller.sal-distributed-datastore - 1.4.1.Boron-SR1 | shard-manager-operational: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-3-shard-default-operational, leaderId=member-4-shard-default-operational, leaderPayloadVersion=5]
2016-11-14 04:32:35,322 | INFO | ult-dispatcher-2 | ShardManager | 195 - org.opendaylight.controller.sal-distributed-datastore - 1.4.1.Boron-SR1 | shard-manager-operational: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-3-shard-entity-ownership-operational, leaderId=member-4-shard-entity-ownership-operational, leaderPayloadVersion=5]
2016-11-14 04:32:35,330 | INFO | ult-dispatcher-5 | ShardManager | 195 - org.opendaylight.controller.sal-distributed-datastore - 1.4.1.Boron-SR1 | shard-manager-operational: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-3-shard-toaster-operational, leaderId=member-2-shard-toaster-operational, leaderPayloadVersion=5]
2016-11-14 04:32:35,331 | INFO | ult-dispatcher-5 | ShardManager | 195 - org.opendaylight.controller.sal-distributed-datastore - 1.4.1.Boron-SR1 | shard-manager-operational: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-3-shard-topology-operational, leaderId=member-2-shard-topology-operational, leaderPayloadVersion=5]
2016-11-14 04:32:35,441 | INFO | lt-dispatcher-16 | ShardManager | 195 - org.opendaylight.controller.sal-distributed-datastore - 1.4.1.Boron-SR1 | shard-manager-config: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-3-shard-inventory-config, leaderId=member-5-shard-inventory-config, leaderPayloadVersion=5]
2016-11-14 04:32:35,441 | INFO | lt-dispatcher-16 | ShardManager | 195 - org.opendaylight.controller.sal-distributed-datastore - 1.4.1.Boron-SR1 | shard-manager-config: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-3-shard-inventory-config, leaderId=null, leaderPayloadVersion=5]
2016-11-14 04:32:35,442 | INFO | lt-dispatcher-18 | ShardManager | 195 - org.opendaylight.controller.sal-distributed-datastore - 1.4.1.Boron-SR1 | shard-manager-config: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-3-shard-toaster-config, leaderId=member-5-shard-toaster-config, leaderPayloadVersion=5]
2016-11-14 04:32:35,442 | INFO | lt-dispatcher-18 | ShardManager | 195 - org.opendaylight.controller.sal-distributed-datastore - 1.4.1.Boron-SR1 | shard-manager-config: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-3-shard-default-config, leaderId=member-2-shard-default-config, leaderPayloadVersion=5]
2016-11-14 04:32:35,442 | INFO | lt-dispatcher-18 | ShardManager | 195 - org.opendaylight.controller.sal-distributed-datastore - 1.4.1.Boron-SR1 | shard-manager-config: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-3-shard-toaster-config, leaderId=null, leaderPayloadVersion=5]
2016-11-14 04:32:35,442 | INFO | lt-dispatcher-18 | ShardManager | 195 - org.opendaylight.controller.sal-distributed-datastore - 1.4.1.Boron-SR1 | shard-manager-config: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-3-shard-topology-config, leaderId=member-5-shard-topology-config, leaderPayloadVersion=5]
2016-11-14 04:32:35,442 | INFO | ult-dispatcher-4 | ShardManager | 195 - org.opendaylight.controller.sal-distributed-datastore - 1.4.1.Boron-SR1 | shard-manager-config: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-3-shard-topology-config, leaderId=null, leaderPayloadVersion=5]
2016-11-14 04:32:35,455 | INFO | lt-dispatcher-16 | ShardManager | 195 - org.opendaylight.controller.sal-distributed-datastore - 1.4.1.Boron-SR1 | shard-manager-config: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-3-shard-default-config, leaderId=null, leaderPayloadVersion=5]