[CONTROLLER-1413] Clustering : Some time one of the rpc (2551) ports not coming up in 2 follower nodes of a 3 node cluster

Thu, 19 Oct 2017 21:26:42 +0000

I am running a 3 node cluster VMs on a Dell Host. I started the cluster after clean start-up, however one of the rpc (2551) ports not coming up in c1 and c3 node ( check below netstat command output for c1 and c3). From Jconsole I checked C2 is the leader.

I executed below features in ordered fashion. And enabled DEBUG logs in akka.conf for all the cluster nodes but put log:set INFO in karaf interface. I see only INFO messages in all log files.

All cluster nodes are reachable to one another.

opendaylight-user@root>feature:install odl-restconf-all
opendaylight-user@root>feature:install odl-mdsal-all
opendaylight-user@root>feature:install odl-openflowplugin-all
opendaylight-user@root>log:set INFO

C1 node
======
Last login: Mon Aug 24 23:26:04 2015 from 10.183.181.12
root@mininet-vm:~# netstat -na | grep 2551
tcp6 0 0 10.183.181.41:2551 :::* LISTEN
tcp6 0 0 10.183.181.41:46156 10.183.181.42:2551 ESTABLISHED
tcp6 0 0 10.183.181.41:2551 10.183.181.43:47001 ESTABLISHED
tcp6 0 0 10.183.181.41:2551 10.183.181.42:46389 ESTABLISHED
root@mininet-vm:~# netstat -na | grep 2550
tcp6 0 0 10.183.181.41:2550 :::* LISTEN
tcp6 0 0 10.183.181.41:54131 10.183.181.43:2550 ESTABLISHED
tcp6 0 0 10.183.181.41:2550 10.183.181.42:59211 ESTABLISHED
tcp6 0 0 10.183.181.41:48564 10.183.181.42:2550 ESTABLISHED
tcp6 0 0 10.183.181.41:2550 10.183.181.43:43979 ESTABLISHED
root@mininet-vm:~#

C2 node
======
root@mininet-vm:~# netstat -na | grep 2550
tcp6 0 0 10.183.181.42:2550 :::* LISTEN
tcp6 0 0 10.183.181.42:2550 10.183.181.41:48564 ESTABLISHED
tcp6 0 0 10.183.181.42:2550 10.183.181.43:50044 ESTABLISHED
tcp6 0 0 10.183.181.42:51749 10.183.181.43:2550 ESTABLISHED
tcp6 0 0 10.183.181.42:59211 10.183.181.41:2550 ESTABLISHED
root@mininet-vm:~# netstat -na | grep 2551
tcp6 0 0 10.183.181.42:2551 :::* LISTEN
tcp6 0 0 10.183.181.42:2551 10.183.181.43:48937 ESTABLISHED
tcp6 0 0 10.183.181.42:2551 10.183.181.41:46156 ESTABLISHED
tcp6 0 0 10.183.181.42:34195 10.183.181.43:2551 ESTABLISHED
tcp6 0 0 10.183.181.42:46389 10.183.181.41:2551 ESTABLISHED

C3 node
======
root@mininet-vm:~# netstat -na | grep 2550
tcp6 0 0 10.183.181.43:2550 :::* LISTEN
tcp6 0 0 10.183.181.43:43979 10.183.181.41:2550 ESTABLISHED
tcp6 467 0 10.183.181.43:2550 10.183.181.42:51749 ESTABLISHED
tcp6 0 0 10.183.181.43:50044 10.183.181.42:2550 ESTABLISHED
tcp6 0 0 10.183.181.43:2550 10.183.181.41:54131 ESTABLISHED
root@mininet-vm:~# netstat -na | grep 2551
tcp6 0 0 10.183.181.43:2551 :::* LISTEN
tcp6 0 0 10.183.181.43:2551 10.183.181.42:34195 ESTABLISHED
tcp6 0 0 10.183.181.43:47001 10.183.181.41:2551 ESTABLISHED
tcp6 0 0 10.183.181.43:48937 10.183.181.42:2551 ESTABLISHED
root@mininet-vm:~#

Please find the attached logs of all cluster nodes.

OpenDaylight JIRA

[CONTROLLER-1413] Clustering : Some time one of the rpc (2551) ports not coming up in 2 follower nodes of a 3 node cluster