Uploaded image for project: 'controller'
  1. controller
  2. CONTROLLER-1412

Clustering : Often odl-cluster-data & odl-cluster-rpc Ports not coming up due config-pusher exception in 3 node cluster setup

    XMLWordPrintable

Details

    • Bug
    • Status: Resolved
    • Resolution: Cannot Reproduce
    • Lithium
    • None
    • clustering
    • None
    • Operating System: All
      Platform: All

    • 4192
    • High

    Description

      I am using a Dell host (32 cores) on which controller 3 VMs are started. each controller has been configured with 6 vCPUs and 6GB RAM. However often I see due to config pusher exception odl-cluster-data (2550) & odl-cluster-rpc Ports (2551) are not coming up in any random node.

      sdnc-host2:~ # c1
      Welcome to Ubuntu 14.04.1 LTS (GNU/Linux 3.13.0-43-generic x86_64)

      Last login: Mon Aug 24 02:15:14 2015 from 10.183.181.12
      root@mininet-vm:~# netstat -na | grep 2551
      root@mininet-vm:~# netstat -na | grep 2550
      root@mininet-vm:~#

      opendaylight-user@root>Exception in thread "config-pusher" java.lang.IllegalStateException: Unable to push configuration due to missing yang models. Required yang models that are missing: [urn:opendaylight:params:xml:ns:yang:controller:netty:threadgroup?module=threadgroup&revision=2013-11-07, urn:opendaylight:params:xml:ns:yang:controller:netty:timer?module=netty-timer&revision=2013-11-19, urn:opendaylight:params:xml:ns:yang:controller:netty:eventexecutor?module=netty-event-executor&revision=2013-11-12]
      at org.opendaylight.controller.netconf.persist.impl.ConfigPusherImpl.getOperationServiceWithRetries(ConfigPusherImpl.java:168)
      at org.opendaylight.controller.netconf.persist.impl.ConfigPusherImpl.pushConfigWithConflictingVersionRetries(ConfigPusherImpl.java:131)
      at org.opendaylight.controller.netconf.persist.impl.ConfigPusherImpl.internalPushConfigs(ConfigPusherImpl.java:103)
      at org.opendaylight.controller.netconf.persist.impl.ConfigPusherImpl.process(ConfigPusherImpl.java:76)
      at org.opendaylight.controller.netconf.persist.impl.osgi.ConfigPersisterActivator$InnerCustomizer$1.run(ConfigPersisterActivator.java:181)
      at java.lang.Thread.run(Thread.java:745)
      Caused by: org.opendaylight.controller.netconf.persist.impl.ConfigPusherImpl$NotEnoughCapabilitiesException: Not enough capabilities for 00-netty.xml(odl-config-netty,odl-config-netty). Expected but not found: [urn:opendaylight:params:xml:ns:yang:controller:netty:threadgroup?module=threadgroup&revision=2013-11-07, urn:opendaylight:params:xml:ns:yang:controller:netty:timer?module=netty-timer&revision=2013-11-19, urn:opendaylight:params:xml:ns:yang:controller:netty:eventexecutor?module=netty-event-executor&revision=2013-11-12]
      at org.opendaylight.controller.netconf.persist.impl.ConfigPusherImpl.getOperationService(ConfigPusherImpl.java:248)
      at org.opendaylight.controller.netconf.persist.impl.ConfigPusherImpl.getOperationServiceWithRetries(ConfigPusherImpl.java:151)
      ... 5 more

      opendaylight-user@root>Exception in thread "config-pusher" java.lang.IllegalStateException: Unable to push configuration due to missing yang models. Required yang models that are missing: [urn:opendaylight:params:xml:ns:yang:controller:md:sal:binding:impl?module=opendaylight-sal-binding-broker-impl&revision=2013-10-28, urn:opendaylight:params:xml:ns:yang:controller:md:sal:binding?module=opendaylight-md-sal-binding&revision=2013-10-28, urn:opendaylight:params:xml:ns:yang:controller:md:sal:dom?module=opendaylight-md-sal-dom&revision=2013-10-28, urn:opendaylight:params:xml:ns:yang:controller:md:sal:dom:impl?module=opendaylight-sal-dom-broker-impl&revision=2013-10-28, urn:opendaylight:params:xml:ns:yang:controller:inmemory-datastore-provider?module=opendaylight-inmemory-datastore-provider&revision=2014-06-17, urn:opendaylight:params:xml:ns:yang:controller:md:sal:common?module=opendaylight-md-sal-common&revision=2013-10-28, urn:opendaylight:params:xml:ns:yang:controller:md:sal:core:spi:operational-dom-store?module=opendaylight-operational-dom-datastore&revision=2014-06-17, urn:opendaylight:params:xml:ns:yang:controller:md:sal:core:spi:config-dom-store?module=opendaylight-config-dom-datastore&revision=2014-06-17, urn:opendaylight:params:xml:ns:yang:controller:md:sal:dom:pingpong?module=opendaylight-pingpong-broker&revision=2014-11-07]
      at org.opendaylight.controller.netconf.persist.impl.ConfigPusherImpl.getOperationServiceWithRetries(ConfigPusherImpl.java:168)
      at org.opendaylight.controller.netconf.persist.impl.ConfigPusherImpl.pushConfigWithConflictingVersionRetries(ConfigPusherImpl.java:131)
      at org.opendaylight.controller.netconf.persist.impl.ConfigPusherImpl.internalPushConfigs(ConfigPusherImpl.java:103)
      at org.opendaylight.controller.netconf.persist.impl.ConfigPusherImpl.process(ConfigPusherImpl.java:76)
      at org.opendaylight.controller.netconf.persist.impl.osgi.ConfigPersisterActivator$InnerCustomizer$1.run(ConfigPersisterActivator.java:181)
      at java.lang.Thread.run(Thread.java:745)
      Caused by: org.opendaylight.controller.netconf.persist.impl.ConfigPusherImpl$NotEnoughCapabilitiesException: Not enough capabilities for 01-md-sal.xml(odl-mdsal-broker,odl-mdsal-broker). Expected but not found: [urn:opendaylight:params:xml:ns:yang:controller:md:sal:binding:impl?module=opendaylight-sal-binding-broker-impl&revision=2013-10-28, urn:opendaylight:params:xml:ns:yang:controller:md:sal:binding?module=opendaylight-md-sal-binding&revision=2013-10-28, urn:opendaylight:params:xml:ns:yang:controller:md:sal:dom?module=opendaylight-md-sal-dom&revision=2013-10-28, urn:opendaylight:params:xml:ns:yang:controller:md:sal:dom:impl?module=opendaylight-sal-dom-broker-impl&revision=2013-10-28, urn:opendaylight:params:xml:ns:yang:controller:inmemory-datastore-provider?module=opendaylight-inmemory-datastore-provider&revision=2014-06-17, urn:opendaylight:params:xml:ns:yang:controller:md:sal:common?module=opendaylight-md-sal-common&revision=2013-10-28, urn:opendaylight:params:xml:ns:yang:controller:md:sal:core:spi:operational-dom-store?module=opendaylight-operational-dom-datastore&revision=2014-06-17, urn:opendaylight:params:xml:ns:yang:controller:md:sal:core:spi:config-dom-store?module=opendaylight-config-dom-datastore&revision=2014-06-17, urn:opendaylight:params:xml:ns:yang:controller:md:sal:dom:pingpong?module=opendaylight-pingpong-broker&revision=2014-11-07]
      at org.opendaylight.controller.netconf.persist.impl.ConfigPusherImpl.getOperationService(ConfigPusherImpl.java:248)
      at org.opendaylight.controller.netconf.persist.impl.ConfigPusherImpl.getOperationServiceWithRetries(ConfigPusherImpl.java:151)
      ... 5 more

      opendaylight-user@root>
      opendaylight-user@root>
      opendaylight-user@root>shutdown -f
      opendaylight-user@root>
      root@mininet-vm:/home/mininet/controller-Li/distributions/karaf/target/assembly/bin#

      3. Another Observation i see:
      High CPU utilization seen during startup and loading of modules

      feature:install odl-restconf-all
      feature:install odl-mdsal-all
      feature:install odl-openflowplugin-all

      sdnc-host2:~ # top
      top - 07:00:23 up 101 days, 17:31, 7 users, load average: 12.17, 9.46, 6.02
      Tasks: 282 total, 1 running, 281 sleeping, 0 stopped, 0 zombie
      Cpu(s): 6.8%us, 0.3%sy, 0.0%ni, 92.9%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
      Mem: 64501M total, 56198M used, 8302M free, 270M buffers
      Swap: 2053M total, 32M used, 2021M free, 32930M cached

      PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
      35009 root 20 0 6550m 3.4g 5868 S 86 5.4 24:11.17 qemu-kvm
      34965 root 20 0 6536m 3.5g 5956 S 63 5.5 22:18.25 qemu-kvm
      34922 root 20 0 6537m 3.4g 5952 S 51 5.4 24:44.43 qemu-kvm
      7396 root 20 0 4317m 1.7g 5284 S 18 2.7 803:54.41 qemu-kvm
      5554 root 20 0 4424m 2.9g 5924 S 6 4.6 367:41.77 qemu-kvm
      120625 root 20 0 16.3g 1.9g 5844 S 2 2.9 208:00.90 qemu-kvm

      Attachments

        1. c5.karaf.rar
          5 kB
          Sanjib Mohapatra

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              Unassigned Unassigned
              sanjib.mohapatra@ericsson.com Sanjib Mohapatra
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: