[CONTROLLER-1473] Node exists in entity-owner data even after killing the instance Created: 18/Jan/16 Updated: 19/Oct/17 Resolved: 09/Sep/16 |
|
| Status: | Resolved |
| Project: | controller |
| Component/s: | clustering |
| Affects Version/s: | Beryllium |
| Fix Version/s: | None |
| Type: | Bug | ||
| Reporter: | Priya Ramasubbu | Assignee: | Unassigned |
| Resolution: | Cannot Reproduce | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Environment: |
Operating System: All |
||
| Issue Links: |
|
||||||||
| External issue ID: | 4992 | ||||||||
| Description |
|
In ovsdb southbound 3 nodes clustering csit, ovsdb-csit-3node-clustering-only-beryllium when a node goes down and if i check the entity owner among the other two nodes Added more delay of 20s to get the entity owner after killing one node. But still the down node exist in operational ds for entity candidates list. Here goes the log for the observed behaviour: KEYWORD ${data} = Utils . Get Data From URI controller@ {controller_index_list}[0], /restconf/operational/entity-owners:entity-ownersDocumentation: Issue a GET request and return the data obtained or on error log the error and fail. Start / End / Elapsed: 20160118 20:06:49.764 / 20160118 20:06:49.792 / 00:00:00.028 00:00:00.020KEYWORD ${response} = RequestsLibrary . Get Request ${session}, ${uri}, ${headers} 00:00:00.005KEYWORD BuiltIn . Return From Keyword If ${response.status_code} == 200, ${response.text} Documentation: Returns from the enclosing user keyword if condition is true. Start / End / Elapsed: 20160118 20:06:49.787 / 20160118 20:06:49.792 / 00:00:00.005 20:06:49.791 INFO Returning from the enclosing user keyword. 20:06:49.792 INFO ${data} = {"entity-owners":{"entity-type":[{"type":"ovsdb","entity":[{"id":"/network-topology:network-topology/network-topology:topology[network-topology:topology-id='ovsdb:1']/network-topology:node[network-top... 00:00:00.001KEYWORD BuiltIn . Log ${data} Documentation: Logs the given message with the given level. Start / End / Elapsed: 20160118 20:06:49.793 / 20160118 20:06:49.794 / 00:00:00.001 20:06:49.793 INFO {"entity-owners":{"entity-type":[{"type":"ovsdb","entity":[{"id":"/network-topology:network-topology/network-topology:topology[network-topology:topology-id='ovsdb:1']/network-topology:node[network-topology:node-id='ovsdb://uuid/a96ec4e2-c457-4a2c-963c-1e6300210032']","candidate":[{"name":"member-1"},{"name":"member-2"},{"name":"member-3"}],"owner":"member-2"}]},{"type":"ovsdb-southbound-provider","entity":[{"id":"/general-entity:entity[general-entity:name='ovsdb-southbound-provider']","candidate":[{"name":"member-1"},{"name":"member-3"},{"name":"member-2"}],"owner":"member-3"}]}]}}KEYWORD ${data} = Utils . Get Data From URI controller@{controller_index_list} [0], /restconf/operational/entity-owners:entity-owners , {"name":"member-1"}, {"name":"member-2"}],"owner":"member-1"}]},{"type":"ovsdb-southbound-provider","entity":[{"id":"/general-entity:entity[general-entity:name='ovsdb-southbound-provider']","candidate":[ {"name":"member-3"}, {"name":"member-1"}, {"name":"member-2"}],"owner":"member-1"}]}]}} Full log is available in below link: |
| Comments |
| Comment by Anil Vishnoi [ 18/Jan/16 ] |
|
It seems to be an issue with the EntityOwnerShipService at clustering level. Moving it to controller/clustering project to get feedback from clustering experts. |
| Comment by Luis Gomez [ 23/Jan/16 ] |
|
This is duplicate of https://bugs.opendaylight.org/show_bug.cgi?id=4992 |
| Comment by Luis Gomez [ 23/Jan/16 ] |
| Comment by Luis Gomez [ 24/Jan/16 ] |
|
This issue is seen in both OpenFlow and OVSDB cluster tests. |
| Comment by Luis Gomez [ 26/Jan/16 ] |
|
Rising priority, it would be good to fix this by Be release. |
| Comment by Moiz Raja [ 26/Jan/16 ] |
|
Controller logs with loglevel set to DEBUG for org.opendaylight.controller.cluster.datastore.entityownership would be useful. Based on the code it seems like we would run into this situation if the switch master was also the leader of the entity-ownership shard. |
| Comment by Tom Pantelis [ 26/Jan/16 ] |
|
Patch https://git.opendaylight.org/gerrit/#/c/26808/ added the code to onPeerDown to remove the downed node as a candidate if the first node is the leader. Previously we didn't remove it b/c of the possibility of network partition as opposed to the process actually being down but 26808 addressed that as well. In onLeaderChanged, it searches for entities owned by the old leader and re-assigns them but it doesn't remove the old leader as candidate. Patch 26808 didn't modify this behavior. When the leader is taken down, if a follower becomes the leader before PeerDown then the old leader gets removed from the candidate list. However if PeerDown occurs before the follower becomes the leader then the old leader candidate isn't removed. The former scenario will occur if the leader is taken down gracefully as it will transfer leadership on shut down. The latter scenario can occur due to timing if the leader process is killed since immediate leadership transfer won't occur. It sounds like the integration test kills the leader so it will be hit or miss if the candidate is removed (due to the 10s election timeout I think PeerDown will occur first most of the time). I created a unit test which illustrates both scenarios. It seems we have a disconnect between the onLeaderChanged and onPeerDown behaviors. In onLeaderChanged, I think we need to remove the old leader as candidate if it's deemed as down (i.e. if PeerDown was previously received). We dot need the logic of re-assigning entities previously owned by the old leader - that's a remnant from prior to 26808. Moiz - what do you think? |
| Comment by Moiz Raja [ 27/Jan/16 ] |
|
That confirms what I was thinking that in this case onPeerDown did not happen on the leader and thus the candidates were not removed. LeaderChanged and onPeerDown can both indicate that a partition has occurred so I agree with you that we should treat both these cases similarly. One thing I would differently is not assume that onPeerDown will happen before LeaderChanged. The reason being that peerDown is controlled by the akka heartbeat configuration and in some cases people may want to set that heartbeat to be greater than the Raft heartbeat and so akka might not be the first to notice the partition. |
| Comment by Tom Pantelis [ 27/Jan/16 ] |
|
The behavior of onPeerDown and onLeaderChanged should be the same and, prior to 26808, they were, i.e. call selectNewOwnerForEntitiesOwnedBy. 26808 should've also changed onLeaderChanged - my bad. I'll push a patch (I have the unit test already). (In reply to Moiz Raja from comment #9) |
| Comment by Tom Pantelis [ 27/Jan/16 ] |
| Comment by Luis Gomez [ 29/Jan/16 ] |
|
Right this works now according to our test suites. |
| Comment by Priya Ramasubbu [ 07/Sep/16 ] |
|
The issue still exists in boron. Current Sandbox link which reproduce the bug again. -Priya |
| Comment by Tom Pantelis [ 07/Sep/16 ] |
|
This is expected behavior now with https://git.opendaylight.org/gerrit/#/c/45271/. On PeerDown, the leader cannot assume the "down" member node is actually down - it might be isolated. Thus we leave it as candidate but re-assign ownership. If the node was actually down, on restart it will first remove itself as a candidate in case no client on the new incarnation registers a candidate. |