[NETCONF-454] Three node cluster does not recover from isolation+ rejoin of a member.

Duplicate — Mon, 22 Jan 2024 21:55:19 +0000

1. Aim:
  Verify that a three node cluster can recover from isolation+rejoin of a member.

1. Description:
  A 3 node cluster was deployed. The netconf test tool was connected and we looked behind the mountpoint from each member of the cluster.
  We then found the leader of the config-topology shard and the config-inventory shard.
  A different memeber was isolated. So in this the leader was member 2. Member-1 was isolated using iptables.
  We looked beyond the mount point of the netconf device from each member of the cluster.
  The isolated member returned HTTP 401 when trying to view the netconf-topology using RESTCONF
  Member-2 ,Member-3 returned HTTP 200 and data when trying to view the netconf-topology and look beyond the mount point.

LOgs are attached. Member-1 was rejoined at approx 04:44

Results when trying to view the netconf-topology after rejoin
Try 1) Member 1: Null Member-2: 404 Member-3: 500
Try 2) Member 1: 200 Member-2: 404 Member-3: 200
Try 3) Member 1: 200 Member-2: 401 Member-3: 200

Results when looking beyond the mount point after rejoin.
Try 1) Member 1: 404 Member-2: 404 Member-3: 500
Try 2) Member 1: 404 Member-2: 404 Member-3: 500
Try 3) Member 1: 404 Member-2: 401 Member-3: 500

1. Test artifact
  Carbon

1. Test setup

Member-1: 10.10.199.105
Member-2: 10.10.199.88
Member-3: 10.10.199.183
Netconf device: 10.10.199.185

1. Steps
  Configure a 3 node cluster
  Start odl on members

Set the log levels for clustering.
log:set DEBUG org.opendaylight.controller.cluster
log:set INFO org.opendaylight.controller.cluster.datastore.node.utils

Features loaded:

odl-jolokia odl-netconf-clustered-topology odl-netconf-connector-all odl-restconf odl-netconf-mdsal

1. REST API CALLS
Find the leader of the shards on DistributedConfigDatastore.

1. 1. Leader of shard-topology-config on DistributedConfigDatastore on each member

curl --request GET \
--url 'http://10.10.199.XXX:8181/jolokia/read/org.opendaylight.controller:Category=Shards,name=member-1-shard-topology-config,type=DistributedConfigDatastore' \
--header 'accept: application/json' \
--header 'authorization: Basic YWRtaW46YWRtaW4=' \
--header 'cache-control: no-cache' \
--header 'content-type: application/json' \
--header 'postman-token: 19ee9a88-1c09-6762-6a54-0b44c8cba583'

Result:

1. 1. Leader of shard-inventory-config on DistributedConfigDatastore on each member

curl --request GET \
--url 'http://10.10.199.XXX:8181/jolokia/read/org.opendaylight.controller:Category=Shards,name=member-1-shard-inventory-config,type=DistributedConfigDatastore' \
--header 'accept: application/json' \
--header 'authorization: Basic YWRtaW46YWRtaW4=' \
--header 'cache-control: no-cache' \
--header 'content-type: application/json' \
--header 'postman-token: 1dc38630-8b41-10e4-cdfa-fb53639c543a'

Result:

1. 1. Confirm no device added to network topology

curl --request GET \
--url http://10.10.199.105:8181/restconf/operational/network-topology:network-topology \
--header 'accept: application/xml' \
--header 'authorization: Basic YWRtaW46YWRtaW4=' \
--header 'cache-control: no-cache' \
--header 'content-type: application/xml' \
--header 'postman-token: 1633d048-98ee-ae9a-00dd-b9f33749cb70'

Result:

1. 1. Start device on testtools-vm

java -Xmx1G -XX:MaxPermSize=256M -Dorg.apache.sshd.registerBouncyCastle=false -jar netconf-testtool-1.2-Carbon-executable.jar --device-count 1 --debug true --schemas-dir ./schemas --md-sal true

1. 1. Add device

curl --request PUT \
--url http://10.10.199.105:8181/restconf/config/network-topology:network-topology/topology/topology-netconf/node/netconf-test-device \
--header 'authorization: Basic YWRtaW46YWRtaW4=' \
--header 'cache-control: no-cache' \
--header 'content-type: application/xml' \
--header 'postman-token: a3868282-fbca-58e4-08ee-fc522a730e30' \
--data ' <node xmlns="urn:TBD:params:xml:ns:yang:network-topology">\r\n <node-id>netconf-test-device</node-id>\r\n <host xmlns="urn:opendaylight:netconf-node-topology">10.10.199.185</host>\r\n <port xmlns="urn:opendaylight:netconf-node-topology">17830</port>\r\n <username xmlns="urn:opendaylight:netconf-node-topology">root</username>\r\n <password xmlns="urn:opendaylight:netconf-node-topology">root</password>\r\n <tcp-only xmlns="urn:opendaylight:netconf-node-topology">false</tcp-only>\r\n <keepalive-delay xmlns="urn:opendaylight:netconf-node-topology">0</keepalive-delay>\r\n </node>\r\n \r\n '

Result:
HTTP 200

1. 1. Confirm device is connected in the operational topology on each of the nodes.

curl --request GET \
--url http://10.10.199.XXX:8181/restconf/operational/network-topology:network-topology \
--header 'accept: application/xml' \
--header 'authorization: Basic YWRtaW46YWRtaW4=' \
--header 'cache-control: no-cache' \
--header 'content-type: application/xml' \
--header 'postman-token: 1633d048-98ee-ae9a-00dd-b9f33749cb70'

Result:
HTTP 200

1. 1. look beyond the mount point from each of the nodes

curl --request GET \
--url http://10.10.199.XXX:8181/restconf/operational/network-topology:network-topology/topology/topology-netconf/node/netconf-test-device/yang-ext:mount \
--header 'accept: application/xml' \
--header 'authorization: Basic YWRtaW46YWRtaW4=' \
--header 'cache-control: no-cache' \
--header 'content-type: application/xml' \
--header 'postman-token: e17517ae-f008-b96e-43ef-960804d7df87'

Result:
HTTP 200

1. 1. Find leader of shard-topology-config on DistributedConfigDatastore on each member after isolation

Member-XXX
curl --request GET \
--url 'http://10.10.199.XXX:8181/jolokia/read/org.opendaylight.controller:Category=Shards,name=member-1-shard-inventory-config,type=DistributedConfigDatastore' \
--header 'accept: application/json' \
--header 'authorization: Basic YWRtaW46YWRtaW4=' \
--header 'cache-control: no-cache' \
--header 'content-type: application/json' \
--header 'postman-token: 607b0a83-e559-f1c6-dfb6-c18fde1c9364'

OpenDaylight JIRA

[NETCONF-454] Three node cluster does not recover from isolation+ rejoin of a member.