[CONTROLLER-1412] Clustering : Often odl-cluster-data & odl-cluster-rpc Ports not coming up due config-pusher exception in 3 node cluster setup Created: 26/Aug/15  Updated: 19/Oct/17  Resolved: 04/Feb/16

Status: Resolved
Project: controller
Component/s: clustering
Affects Version/s: Lithium
Fix Version/s: None

Type: Bug
Reporter: Sanjib Mohapatra Assignee: Unassigned
Resolution: Cannot Reproduce Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Operating System: All
Platform: All

Attachments: File c5.karaf.rar    
Issue Links:
Blocks
is blocked by NETCONF-32 Feature install of odl-netconf-connec... Verified
External issue ID: 4192
Priority: High

 Description   

I am using a Dell host (32 cores) on which controller 3 VMs are started. each controller has been configured with 6 vCPUs and 6GB RAM. However often I see due to config pusher exception odl-cluster-data (2550) & odl-cluster-rpc Ports (2551) are not coming up in any random node.

sdnc-host2:~ # c1
Welcome to Ubuntu 14.04.1 LTS (GNU/Linux 3.13.0-43-generic x86_64)

Last login: Mon Aug 24 02:15:14 2015 from 10.183.181.12
root@mininet-vm:~# netstat -na | grep 2551
root@mininet-vm:~# netstat -na | grep 2550
root@mininet-vm:~#

opendaylight-user@root>Exception in thread "config-pusher" java.lang.IllegalStateException: Unable to push configuration due to missing yang models. Required yang models that are missing: [urn:opendaylight:params:xml:ns:yang:controller:netty:threadgroup?module=threadgroup&revision=2013-11-07, urn:opendaylight:params:xml:ns:yang:controller:netty:timer?module=netty-timer&revision=2013-11-19, urn:opendaylight:params:xml:ns:yang:controller:netty:eventexecutor?module=netty-event-executor&revision=2013-11-12]
at org.opendaylight.controller.netconf.persist.impl.ConfigPusherImpl.getOperationServiceWithRetries(ConfigPusherImpl.java:168)
at org.opendaylight.controller.netconf.persist.impl.ConfigPusherImpl.pushConfigWithConflictingVersionRetries(ConfigPusherImpl.java:131)
at org.opendaylight.controller.netconf.persist.impl.ConfigPusherImpl.internalPushConfigs(ConfigPusherImpl.java:103)
at org.opendaylight.controller.netconf.persist.impl.ConfigPusherImpl.process(ConfigPusherImpl.java:76)
at org.opendaylight.controller.netconf.persist.impl.osgi.ConfigPersisterActivator$InnerCustomizer$1.run(ConfigPersisterActivator.java:181)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.opendaylight.controller.netconf.persist.impl.ConfigPusherImpl$NotEnoughCapabilitiesException: Not enough capabilities for 00-netty.xml(odl-config-netty,odl-config-netty). Expected but not found: [urn:opendaylight:params:xml:ns:yang:controller:netty:threadgroup?module=threadgroup&revision=2013-11-07, urn:opendaylight:params:xml:ns:yang:controller:netty:timer?module=netty-timer&revision=2013-11-19, urn:opendaylight:params:xml:ns:yang:controller:netty:eventexecutor?module=netty-event-executor&revision=2013-11-12]
at org.opendaylight.controller.netconf.persist.impl.ConfigPusherImpl.getOperationService(ConfigPusherImpl.java:248)
at org.opendaylight.controller.netconf.persist.impl.ConfigPusherImpl.getOperationServiceWithRetries(ConfigPusherImpl.java:151)
... 5 more

opendaylight-user@root>Exception in thread "config-pusher" java.lang.IllegalStateException: Unable to push configuration due to missing yang models. Required yang models that are missing: [urn:opendaylight:params:xml:ns:yang:controller:md:sal:binding:impl?module=opendaylight-sal-binding-broker-impl&revision=2013-10-28, urn:opendaylight:params:xml:ns:yang:controller:md:sal:binding?module=opendaylight-md-sal-binding&revision=2013-10-28, urn:opendaylight:params:xml:ns:yang:controller:md:sal:dom?module=opendaylight-md-sal-dom&revision=2013-10-28, urn:opendaylight:params:xml:ns:yang:controller:md:sal:dom:impl?module=opendaylight-sal-dom-broker-impl&revision=2013-10-28, urn:opendaylight:params:xml:ns:yang:controller:inmemory-datastore-provider?module=opendaylight-inmemory-datastore-provider&revision=2014-06-17, urn:opendaylight:params:xml:ns:yang:controller:md:sal:common?module=opendaylight-md-sal-common&revision=2013-10-28, urn:opendaylight:params:xml:ns:yang:controller:md:sal:core:spi:operational-dom-store?module=opendaylight-operational-dom-datastore&revision=2014-06-17, urn:opendaylight:params:xml:ns:yang:controller:md:sal:core:spi:config-dom-store?module=opendaylight-config-dom-datastore&revision=2014-06-17, urn:opendaylight:params:xml:ns:yang:controller:md:sal:dom:pingpong?module=opendaylight-pingpong-broker&revision=2014-11-07]
at org.opendaylight.controller.netconf.persist.impl.ConfigPusherImpl.getOperationServiceWithRetries(ConfigPusherImpl.java:168)
at org.opendaylight.controller.netconf.persist.impl.ConfigPusherImpl.pushConfigWithConflictingVersionRetries(ConfigPusherImpl.java:131)
at org.opendaylight.controller.netconf.persist.impl.ConfigPusherImpl.internalPushConfigs(ConfigPusherImpl.java:103)
at org.opendaylight.controller.netconf.persist.impl.ConfigPusherImpl.process(ConfigPusherImpl.java:76)
at org.opendaylight.controller.netconf.persist.impl.osgi.ConfigPersisterActivator$InnerCustomizer$1.run(ConfigPersisterActivator.java:181)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.opendaylight.controller.netconf.persist.impl.ConfigPusherImpl$NotEnoughCapabilitiesException: Not enough capabilities for 01-md-sal.xml(odl-mdsal-broker,odl-mdsal-broker). Expected but not found: [urn:opendaylight:params:xml:ns:yang:controller:md:sal:binding:impl?module=opendaylight-sal-binding-broker-impl&revision=2013-10-28, urn:opendaylight:params:xml:ns:yang:controller:md:sal:binding?module=opendaylight-md-sal-binding&revision=2013-10-28, urn:opendaylight:params:xml:ns:yang:controller:md:sal:dom?module=opendaylight-md-sal-dom&revision=2013-10-28, urn:opendaylight:params:xml:ns:yang:controller:md:sal:dom:impl?module=opendaylight-sal-dom-broker-impl&revision=2013-10-28, urn:opendaylight:params:xml:ns:yang:controller:inmemory-datastore-provider?module=opendaylight-inmemory-datastore-provider&revision=2014-06-17, urn:opendaylight:params:xml:ns:yang:controller:md:sal:common?module=opendaylight-md-sal-common&revision=2013-10-28, urn:opendaylight:params:xml:ns:yang:controller:md:sal:core:spi:operational-dom-store?module=opendaylight-operational-dom-datastore&revision=2014-06-17, urn:opendaylight:params:xml:ns:yang:controller:md:sal:core:spi:config-dom-store?module=opendaylight-config-dom-datastore&revision=2014-06-17, urn:opendaylight:params:xml:ns:yang:controller:md:sal:dom:pingpong?module=opendaylight-pingpong-broker&revision=2014-11-07]
at org.opendaylight.controller.netconf.persist.impl.ConfigPusherImpl.getOperationService(ConfigPusherImpl.java:248)
at org.opendaylight.controller.netconf.persist.impl.ConfigPusherImpl.getOperationServiceWithRetries(ConfigPusherImpl.java:151)
... 5 more

opendaylight-user@root>
opendaylight-user@root>
opendaylight-user@root>shutdown -f
opendaylight-user@root>
root@mininet-vm:/home/mininet/controller-Li/distributions/karaf/target/assembly/bin#

3. Another Observation i see:
High CPU utilization seen during startup and loading of modules

feature:install odl-restconf-all
feature:install odl-mdsal-all
feature:install odl-openflowplugin-all

sdnc-host2:~ # top
top - 07:00:23 up 101 days, 17:31, 7 users, load average: 12.17, 9.46, 6.02
Tasks: 282 total, 1 running, 281 sleeping, 0 stopped, 0 zombie
Cpu(s): 6.8%us, 0.3%sy, 0.0%ni, 92.9%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 64501M total, 56198M used, 8302M free, 270M buffers
Swap: 2053M total, 32M used, 2021M free, 32930M cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
35009 root 20 0 6550m 3.4g 5868 S 86 5.4 24:11.17 qemu-kvm
34965 root 20 0 6536m 3.5g 5956 S 63 5.5 22:18.25 qemu-kvm
34922 root 20 0 6537m 3.4g 5952 S 51 5.4 24:44.43 qemu-kvm
7396 root 20 0 4317m 1.7g 5284 S 18 2.7 803:54.41 qemu-kvm
5554 root 20 0 4424m 2.9g 5924 S 6 4.6 367:41.77 qemu-kvm
120625 root 20 0 16.3g 1.9g 5844 S 2 2.9 208:00.90 qemu-kvm

 Comments   
Comment by Moiz Raja [ 27/Oct/15 ]

Is this reproducible on Lithium?

Comment by Sanjib Mohapatra [ 28/Oct/15 ]

I have not seen it in SR1 Lithium.

Comment by Sanjib Mohapatra [ 03/Nov/15 ]

c5.karaf.log contains below ERROR message

2015-11-03 06:13:22,029 | ERROR | config-pusher | ConfigPusherImpl | 143 - org.opendaylight.controller.config-persister-impl - 0.3.2.Lithium-SR2 | Unable to push configuration due to missing yang models. Yang models that are missing, but required by the configuration: [urn:opendaylight:params:xml:ns:yang:controller:netty:timer?module=netty-timer&revision=2013-11-19]. For each mentioned model check: 1. that the mentioned yang model namespace/name/revision is identical to those in the yang model itself 2. the yang file is present in the system 3. the bundle with that yang file is present in the system and active 4. the yang parser did not fail while attempting to parse that model

Comment by Sanjib Mohapatra [ 03/Nov/15 ]

Attachment c5.karaf.rar has been added with description: c5.karaf.log

Comment by Sanjib Mohapatra [ 03/Nov/15 ]

Tested in in SR2 in 5 node cluster, finding the issue often, Please refer my previous comment with log attachment.

Comment by Tom Pantelis [ 04/Feb/16 ]

This is not an issue with clustering. The 00-netty module is failing to load due to a missing netty-timer yang model which prevents every other config module, including clustering, from loading. I don't know why the yang model wasn't found in your SR2 setup - I haven't heard of this specific issue elsewhere. I suspect a timing issue where the bundle containing the netty-timer yang model hadn't yet activated when it tried to push the config. There were improvements in this area in Be.

I'm going to close this bug since it's not related to clustering. If the issue persists in Be or the current Li release then open another bug (I would say the config component) and/or ask on the controller-dev mailing list.

Generated at Wed Feb 07 19:55:29 UTC 2024 using Jira 8.20.10#820010-sha1:ace47f9899e9ee25d7157d59aa17ab06aee30d3d.