Requirement :  ODL clustering for high availability (HA) on data distribution 

Failover/High Availability failing in ODL cluster:

• Expected : In case any of the ODL instance gets down/restarted due to network splits or internal error, other instance in cluster should be available and functional. If the affected instance is having master mount, the other instance who is elected as master by re-election should be able to re-register the devices and resume the operations. Once the affected instance comes up, it should be able to join the cluster as member node and register the slave mounts. **
• Observation : When the odl instance which is having the master mount restarts, election happens among the other node in the cluster and elects the new leader. Now the new leader is trying to re-register the master mount but failed at a point due to the termination of the Akka Cluster Singleton Actor. Hence the cluster goes to idle state and failed to assign owner for the device DOM entity. In this case, the configuration of already mounted device/ new mounts will fail.  

Env Configuration:

• 3 node k8s Cluster ( 1 master & 3 worker nodes) with 3 ODL instances running on each node
• CPU :  8 Cores
• RAM : 20GB
• Java Heap size : Min – 512MB Max – 16GB
• JDK version : 11
• Kubernetes version : 1.19.1
• Docker version : 20.10.7

ODL features installed to enable clustering:

• odl-netconf-clustered-topology
• odl-restconf-all

Device configured : Netconf devices , all devices having same schema(tested with 250 devices)