[CONTROLLER-1412] Clustering : Often odl-cluster-data & odl-cluster-rpc Ports not coming up due config-pusher exception in 3 node cluster setup

Blocks — Thu, 19 Oct 2017 21:26:52 +0000

I am using a Dell host (32 cores) on which controller 3 VMs are started. each controller has been configured with 6 vCPUs and 6GB RAM. However often I see due to config pusher exception odl-cluster-data (2550) & odl-cluster-rpc Ports (2551) are not coming up in any random node.

sdnc-host2:~ # c1
Welcome to Ubuntu 14.04.1 LTS (GNU/Linux 3.13.0-43-generic x86_64)

Documentation: https://help.ubuntu.com/

Last login: Mon Aug 24 02:15:14 2015 from 10.183.181.12
root@mininet-vm:~# netstat -na | grep 2551
root@mininet-vm:~# netstat -na | grep 2550
root@mininet-vm:~#

opendaylight-user@root>Exception in thread "config-pusher" java.lang.IllegalStateException: Unable to push configuration due to missing yang models. Required yang models that are missing: [urn:opendaylight:params:xml:ns:yang:controller:md:sal:binding:impl?module=opendaylight-sal-binding-broker-impl&revision=2013-10-28, urn:opendaylight:params:xml:ns:yang:controller:md:sal:binding?module=opendaylight-md-sal-binding&revision=2013-10-28, urn:opendaylight:params:xml:ns:yang:controller:md:sal:dom?module=opendaylight-md-sal-dom&revision=2013-10-28, urn:opendaylight:params:xml:ns:yang:controller:md:sal:dom:impl?module=opendaylight-sal-dom-broker-impl&revision=2013-10-28, urn:opendaylight:params:xml:ns:yang:controller:inmemory-datastore-provider?module=opendaylight-inmemory-datastore-provider&revision=2014-06-17, urn:opendaylight:params:xml:ns:yang:controller:md:sal:common?module=opendaylight-md-sal-common&revision=2013-10-28, urn:opendaylight:params:xml:ns:yang:controller:md:sal:core:spi:operational-dom-store?module=opendaylight-operational-dom-datastore&revision=2014-06-17, urn:opendaylight:params:xml:ns:yang:controller:md:sal:core:spi:config-dom-store?module=opendaylight-config-dom-datastore&revision=2014-06-17, urn:opendaylight:params:xml:ns:yang:controller:md:sal:dom:pingpong?module=opendaylight-pingpong-broker&revision=2014-11-07]
at org.opendaylight.controller.netconf.persist.impl.ConfigPusherImpl.getOperationServiceWithRetries(ConfigPusherImpl.java:168)
at org.opendaylight.controller.netconf.persist.impl.ConfigPusherImpl.pushConfigWithConflictingVersionRetries(ConfigPusherImpl.java:131)
at org.opendaylight.controller.netconf.persist.impl.ConfigPusherImpl.internalPushConfigs(ConfigPusherImpl.java:103)
at org.opendaylight.controller.netconf.persist.impl.ConfigPusherImpl.process(ConfigPusherImpl.java:76)
at org.opendaylight.controller.netconf.persist.impl.osgi.ConfigPersisterActivator$InnerCustomizer$1.run(ConfigPersisterActivator.java:181)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.opendaylight.controller.netconf.persist.impl.ConfigPusherImpl$NotEnoughCapabilitiesException: Not enough capabilities for 01-md-sal.xml(odl-mdsal-broker,odl-mdsal-broker). Expected but not found: [urn:opendaylight:params:xml:ns:yang:controller:md:sal:binding:impl?module=opendaylight-sal-binding-broker-impl&revision=2013-10-28, urn:opendaylight:params:xml:ns:yang:controller:md:sal:binding?module=opendaylight-md-sal-binding&revision=2013-10-28, urn:opendaylight:params:xml:ns:yang:controller:md:sal:dom?module=opendaylight-md-sal-dom&revision=2013-10-28, urn:opendaylight:params:xml:ns:yang:controller:md:sal:dom:impl?module=opendaylight-sal-dom-broker-impl&revision=2013-10-28, urn:opendaylight:params:xml:ns:yang:controller:inmemory-datastore-provider?module=opendaylight-inmemory-datastore-provider&revision=2014-06-17, urn:opendaylight:params:xml:ns:yang:controller:md:sal:common?module=opendaylight-md-sal-common&revision=2013-10-28, urn:opendaylight:params:xml:ns:yang:controller:md:sal:core:spi:operational-dom-store?module=opendaylight-operational-dom-datastore&revision=2014-06-17, urn:opendaylight:params:xml:ns:yang:controller:md:sal:core:spi:config-dom-store?module=opendaylight-config-dom-datastore&revision=2014-06-17, urn:opendaylight:params:xml:ns:yang:controller:md:sal:dom:pingpong?module=opendaylight-pingpong-broker&revision=2014-11-07]
at org.opendaylight.controller.netconf.persist.impl.ConfigPusherImpl.getOperationService(ConfigPusherImpl.java:248)
at org.opendaylight.controller.netconf.persist.impl.ConfigPusherImpl.getOperationServiceWithRetries(ConfigPusherImpl.java:151)
... 5 more

opendaylight-user@root>
opendaylight-user@root>
opendaylight-user@root>shutdown -f
opendaylight-user@root>
root@mininet-vm:/home/mininet/controller-Li/distributions/karaf/target/assembly/bin#

3. Another Observation i see:
High CPU utilization seen during startup and loading of modules

feature:install odl-restconf-all
feature:install odl-mdsal-all
feature:install odl-openflowplugin-all

sdnc-host2:~ # top
top - 07:00:23 up 101 days, 17:31, 7 users, load average: 12.17, 9.46, 6.02
Tasks: 282 total, 1 running, 281 sleeping, 0 stopped, 0 zombie
Cpu(s): 6.8%us, 0.3%sy, 0.0%ni, 92.9%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 64501M total, 56198M used, 8302M free, 270M buffers
Swap: 2053M total, 32M used, 2021M free, 32930M cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
35009 root 20 0 6550m 3.4g 5868 S 86 5.4 24:11.17 qemu-kvm
34965 root 20 0 6536m 3.5g 5956 S 63 5.5 22:18.25 qemu-kvm
34922 root 20 0 6537m 3.4g 5952 S 51 5.4 24:44.43 qemu-kvm
7396 root 20 0 4317m 1.7g 5284 S 18 2.7 803:54.41 qemu-kvm
5554 root 20 0 4424m 2.9g 5924 S 6 4.6 367:41.77 qemu-kvm
120625 root 20 0 16.3g 1.9g 5844 S 2 2.9 208:00.90 qemu-kvm

OpenDaylight JIRA

[CONTROLLER-1412] Clustering : Often odl-cluster-data & odl-cluster-rpc Ports not coming up due config-pusher exception in 3 node cluster setup