-
- Aim:
Verify that a three node cluster can recover from isolation+rejoin of a member.
- Aim:
-
- Description:
A 3 node cluster was deployed. The netconf test tool was connected and we looked behind the mountpoint from each member of the cluster.
We then found the leader of the config-topology shard and the config-inventory shard.
A different memeber was isolated. So in this the leader was member 2. Member-1 was isolated using iptables.
We looked beyond the mount point of the netconf device from each member of the cluster.
The isolated member returned HTTP 401 when trying to view the netconf-topology using RESTCONF
Member-2 ,Member-3 returned HTTP 200 and data when trying to view the netconf-topology and look beyond the mount point.
- Description:
LOgs are attached. Member-1 was rejoined at approx 04:44
Results when trying to view the netconf-topology after rejoin
Try 1) Member 1: Null Member-2: 404 Member-3: 500
Try 2) Member 1: 200 Member-2: 404 Member-3: 200
Try 3) Member 1: 200 Member-2: 401 Member-3: 200
Results when looking beyond the mount point after rejoin.
Try 1) Member 1: 404 Member-2: 404 Member-3: 500
Try 2) Member 1: 404 Member-2: 404 Member-3: 500
Try 3) Member 1: 404 Member-2: 401 Member-3: 500
-
- Test artifact
Carbon
- Test artifact
-
- Test setup
Member-1: 10.10.199.105
Member-2: 10.10.199.88
Member-3: 10.10.199.183
Netconf device: 10.10.199.185
-
- Steps
Configure a 3 node cluster
Start odl on members
- Steps
Set the log levels for clustering.
log:set DEBUG org.opendaylight.controller.cluster
log:set INFO org.opendaylight.controller.cluster.datastore.node.utils
- Features loaded:
odl-jolokia odl-netconf-clustered-topology odl-netconf-connector-all odl-restconf odl-netconf-mdsal
-
- REST API CALLS
- Find the leader of the shards on DistributedConfigDatastore.
-
-
- Leader of shard-topology-config on DistributedConfigDatastore on each member
-
curl --request GET \
--url 'http://10.10.199.XXX:8181/jolokia/read/org.opendaylight.controller:Category=Shards,name=member-1-shard-topology-config,type=DistributedConfigDatastore' \
--header 'accept: application/json' \
--header 'authorization: Basic YWRtaW46YWRtaW4=' \
--header 'cache-control: no-cache' \
--header 'content-type: application/json' \
--header 'postman-token: 19ee9a88-1c09-6762-6a54-0b44c8cba583'
Result:
-
-
- Leader of shard-inventory-config on DistributedConfigDatastore on each member
-
curl --request GET \
--url 'http://10.10.199.XXX:8181/jolokia/read/org.opendaylight.controller:Category=Shards,name=member-1-shard-inventory-config,type=DistributedConfigDatastore' \
--header 'accept: application/json' \
--header 'authorization: Basic YWRtaW46YWRtaW4=' \
--header 'cache-control: no-cache' \
--header 'content-type: application/json' \
--header 'postman-token: 1dc38630-8b41-10e4-cdfa-fb53639c543a'
Result:
-
-
- Confirm no device added to network topology
-
curl --request GET \
--url http://10.10.199.105:8181/restconf/operational/network-topology:network-topology \
--header 'accept: application/xml' \
--header 'authorization: Basic YWRtaW46YWRtaW4=' \
--header 'cache-control: no-cache' \
--header 'content-type: application/xml' \
--header 'postman-token: 1633d048-98ee-ae9a-00dd-b9f33749cb70'
Result:
-
-
- Start device on testtools-vm
-
java -Xmx1G -XX:MaxPermSize=256M -Dorg.apache.sshd.registerBouncyCastle=false -jar netconf-testtool-1.2-Carbon-executable.jar --device-count 1 --debug true --schemas-dir ./schemas --md-sal true
-
-
- Add device
-
curl --request PUT \
--url http://10.10.199.105:8181/restconf/config/network-topology:network-topology/topology/topology-netconf/node/netconf-test-device \
--header 'authorization: Basic YWRtaW46YWRtaW4=' \
--header 'cache-control: no-cache' \
--header 'content-type: application/xml' \
--header 'postman-token: a3868282-fbca-58e4-08ee-fc522a730e30' \
--data ' <node xmlns="urn:TBD:params:xml:ns:yang:network-topology">\r\n <node-id>netconf-test-device</node-id>\r\n <host xmlns="urn:opendaylight:netconf-node-topology">10.10.199.185</host>\r\n <port xmlns="urn:opendaylight:netconf-node-topology">17830</port>\r\n <username xmlns="urn:opendaylight:netconf-node-topology">root</username>\r\n <password xmlns="urn:opendaylight:netconf-node-topology">root</password>\r\n <tcp-only xmlns="urn:opendaylight:netconf-node-topology">false</tcp-only>\r\n <keepalive-delay xmlns="urn:opendaylight:netconf-node-topology">0</keepalive-delay>\r\n </node>\r\n \r\n '
Result:
HTTP 200
-
-
- Confirm device is connected in the operational topology on each of the nodes.
-
curl --request GET \
--url http://10.10.199.XXX:8181/restconf/operational/network-topology:network-topology \
--header 'accept: application/xml' \
--header 'authorization: Basic YWRtaW46YWRtaW4=' \
--header 'cache-control: no-cache' \
--header 'content-type: application/xml' \
--header 'postman-token: 1633d048-98ee-ae9a-00dd-b9f33749cb70'
Result:
HTTP 200
-
-
- look beyond the mount point from each of the nodes
-
curl --request GET \
--url http://10.10.199.XXX:8181/restconf/operational/network-topology:network-topology/topology/topology-netconf/node/netconf-test-device/yang-ext:mount \
--header 'accept: application/xml' \
--header 'authorization: Basic YWRtaW46YWRtaW4=' \
--header 'cache-control: no-cache' \
--header 'content-type: application/xml' \
--header 'postman-token: e17517ae-f008-b96e-43ef-960804d7df87'
Result:
HTTP 200
-
-
- Find leader of shard-topology-config on DistributedConfigDatastore on each member after isolation
-
Member-XXX
curl --request GET \
--url 'http://10.10.199.XXX:8181/jolokia/read/org.opendaylight.controller:Category=Shards,name=member-1-shard-inventory-config,type=DistributedConfigDatastore' \
--header 'accept: application/json' \
--header 'authorization: Basic YWRtaW46YWRtaW4=' \
--header 'cache-control: no-cache' \
--header 'content-type: application/json' \
--header 'postman-token: 607b0a83-e559-f1c6-dfb6-c18fde1c9364'
- is duplicated by
-
CONTROLLER-1784 DataTreeChangeListener when registered during isolation (possibly) does not receive notifications even after rejoin
- Resolved
- relates to
-
CONTROLLER-1992 Expose entity ownership information via MD-SAL
- Resolved
-
NETCONF-1039 Remove RemoteDeviceConnector class
- Resolved