https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/CONTROLLER-1269 controller <p>Started Helium SR-3 downloaded as a prebuilt tarball from the download page. Installed the following features and got the attached exception</p> <p>feature:install odl-base-all<br/> feature:install odl-netconf-connector-all<br/> feature:install odl-config-all<br/> feature:install odl-restconf-all</p> <p>Operating System: All<br/> Platform: All</p> CONTROLLER-1269

Exception in thread "config-pusher"

Bug Resolved Done Unassigned Brian Kaczynski Tue, 21 Apr 2015 16:25:05 +0000 Thu, 19 Oct 2017 21:29:16 +0000 Fri, 13 Nov 2015 15:15:52 +0000 Helium config 0 5 <p>Attachment file_3034.txt has been added with description: Stack trace of exception</p> <p>Here's a version of the exception from the attachment with a bunch of the data from the first line trimmed to get it down to something reasonable to git a glance at.</p> <p>Exception in thread "config-pusher" java.lang.IllegalStateException: Failed to send edit-config for configuration ConfigSnapshot</p> {configSnapshot='...', capabilities=[...]} <p> at org.opendaylight.controller.netconf.persist.impl.ConfigPusherImpl.sendRequestGetResponseCheckIsOK(ConfigPusherImpl.java:293)<br/> at org.opendaylight.controller.netconf.persist.impl.ConfigPusherImpl.pushConfig(ConfigPusherImpl.java:244)<br/> at org.opendaylight.controller.netconf.persist.impl.ConfigPusherImpl.pushConfigWithConflictingVersionRetries(ConfigPusherImpl.java:136)<br/> at org.opendaylight.controller.netconf.persist.impl.ConfigPusherImpl.internalPushConfigs(ConfigPusherImpl.java:109)<br/> at org.opendaylight.controller.netconf.persist.impl.ConfigPusherImpl.process(ConfigPusherImpl.java:76)<br/> at org.opendaylight.controller.netconf.persist.impl.osgi.ConfigPersisterActivator$InnerCustomizer$1.run(ConfigPersisterActivator.java:178)<br/> at java.lang.Thread.run(Thread.java:745)<br/> Caused by: NetconfDocumentedException{message=Unable to save ref name authz-connector-default for instance org.opendaylight.controller:instanceName=aaa-authz-service,TransactionName=ConfigTransaction-28-29,type=Module,moduleFactoryName=aaa-authz-service, errorType=application, errorTag=operation_failed, errorSeverity=error, errorInfo={}}<br/> at org.opendaylight.controller.netconf.confignetconfconnector.operations.editconfig.EditConfig.setServicesOnTransaction(EditConfig.java:164)<br/> at org.opendaylight.controller.netconf.confignetconfconnector.operations.editconfig.EditConfig.test(EditConfig.java:120)<br/> at org.opendaylight.controller.netconf.confignetconfconnector.operations.editconfig.EditConfig.executeTests(EditConfig.java:97)<br/> at org.opendaylight.controller.netconf.confignetconfconnector.operations.editconfig.EditConfig.getResponseInternal(EditConfig.java:76)<br/> at org.opendaylight.controller.netconf.confignetconfconnector.operations.editconfig.EditConfig.handleWithNoSubsequentOperations(EditConfig.java:307)<br/> at org.opendaylight.controller.netconf.util.mapping.AbstractLastNetconfOperation.handle(AbstractLastNetconfOperation.java:33)<br/> at org.opendaylight.controller.netconf.util.mapping.AbstractNetconfOperation.handle(AbstractNetconfOperation.java:105)<br/> at org.opendaylight.controller.netconf.persist.impl.ConfigPusherImpl.sendRequestGetResponseCheckIsOK(ConfigPusherImpl.java:288)<br/> ... 6 more</p> <p>Are you using the aaa-authz service?</p> <p>I am not sure if that is a tested feature in Helium. Could you try without it?</p> <p>Also, can you send the output of feature:list -i to give an idea of what features you have installed?</p> <p>&gt; feature:install odl-base-all<br/> &gt; feature:install odl-netconf-connector-all<br/> &gt; feature:install odl-config-all<br/> &gt; feature:install odl-restconf-all</p> <p>Does this happen even if odl-netconf-connector-all is installed last?<br/> (I do not really expect this bug to be caused by netconf-connector not getting updated schemas, but it is safer to test for it anyway.)</p> <p>Vratko - Tried putting odl-netconf-connector-all last, still experienced the exception. </p> <p>Ed - I had only typed the feature:install commands for the features I listed, but odl-aaa got installed somehow. Is there a way to install those features without getting that pulled in? Should I uninstall the feature and restart the controller to see if it comes back?</p> <p>Here's the output of feature:list | grep " x ":<br/> odl-aaa-authn | 0.1.3-Helium-SR3 | x | odl-aaa-0.1.3-Helium-SR3 | OpenDaylight :: AAA :: Authentication <br/> odl-aaa-authn-plugin | 0.1.3-Helium-SR3 | x | odl-aaa-0.1.3-Helium-SR3 | OpenDaylight :: AAA :: ODL NETCONF Plugin <br/> standard | 3.0.1 | x | standard-3.0.1 | Karaf standard feature <br/> config | 3.0.1 | x | standard-3.0.1 | Provide OSGi ConfigAdmin support <br/> region | 3.0.1 | x | standard-3.0.1 | Provide Region Support <br/> package | 3.0.1 | x | standard-3.0.1 | Package commands and mbeans <br/> http | 3.0.1 | x | standard-3.0.1 | Implementation of the OSGI HTTP Service <br/> war | 3.0.1 | x | standard-3.0.1 | Turn Karaf as a full WebContainer <br/> kar | 3.0.1 | x | standard-3.0.1 | Provide KAR (KARaf archive) support <br/> ssh | 3.0.1 | x | standard-3.0.1 | Provide a SSHd server on Karaf <br/> management | 3.0.1 | x | standard-3.0.1 | Provide a JMX MBeanServer and a set of MBeans in K<br/> odl-netconf-connector-all | 1.1.3-Helium-SR3 | x | odl-controller-1.1.3-Helium-SR3 | OpenDaylight :: Netconf Connector :: All <br/> odl-netconf-connector | 1.1.3-Helium-SR3 | x | odl-controller-1.1.3-Helium-SR3 | OpenDaylight :: Netconf Connector :: Netconf Conne<br/> odl-netconf-connector-ssh | 1.1.3-Helium-SR3 | x | odl-controller-1.1.3-Helium-SR3 | OpenDaylight :: Netconf Connector :: Netconf Conne<br/> odl-netconf-ssh | 0.2.8-Helium-SR3 | x | odl-controller-1.1.3-Helium-SR3 | OpenDaylight :: Netconf Connector :: SSH <br/> odl-netconf-tcp | 0.2.8-Helium-SR3 | x | odl-controller-1.1.3-Helium-SR3 | OpenDaylight :: Netconf Connector :: TCP <br/> odl-config-all | 0.2.8-Helium-SR3 | x | odl-config-0.2.8-Helium-SR3 | OpenDaylight :: Config :: All <br/> odl-mdsal-common | 1.1.3-Helium-SR3 | x | odl-config-0.2.8-Helium-SR3 | OpenDaylight :: Config :: All <br/> odl-config-api | 0.2.8-Helium-SR3 | x | odl-config-0.2.8-Helium-SR3 | OpenDaylight :: Config :: API <br/> odl-config-netty-config-api | 0.2.8-Helium-SR3 | x | odl-config-0.2.8-Helium-SR3 | OpenDaylight :: Config :: Netty Config API <br/> odl-config-core | 0.2.8-Helium-SR3 | x | odl-config-0.2.8-Helium-SR3 | OpenDaylight :: Config :: Core <br/> odl-config-manager | 0.2.8-Helium-SR3 | x | odl-config-0.2.8-Helium-SR3 | OpenDaylight :: Config :: Manager <br/> odl-config-netty | 0.2.8-Helium-SR3 | x | odl-config-persister-0.2.8-Helium-SR3 | OpenDaylight :: Config-Netty <br/> odl-base-all | 1.4.5-Helium-SR3 | x | odl-base-1.4.5-Helium-SR3 | OpenDaylight Controller <br/> odl-base-dummy-console | 1.1.3-Helium-SR3 | x | odl-base-1.4.5-Helium-SR3 | Temporary Dummy Console <br/> odl-base-felix-dm | 3.1.0 | x | odl-base-1.4.5-Helium-SR3 | Felix Dependency Manager <br/> odl-base-aries-spi-fly | 1.0.0 | x | odl-base-1.4.5-Helium-SR3 | Aries SPI Fly <br/> odl-base-netty | 4.0.23.Final | x | odl-base-1.4.5-Helium-SR3 | <br/> odl-base-jersey | 1.17 | x | odl-base-1.4.5-Helium-SR3 | Jersey <br/> odl-base-jackson | 2.3.2 | x | odl-base-1.4.5-Helium-SR3 | Jackson JAX-RS <br/> odl-base-slf4j | 1.7.2 | x | odl-base-1.4.5-Helium-SR3 | SLF4J Logging <br/> odl-base-apache-commons | 1.4.5-Helium-SR3 | x | odl-base-1.4.5-Helium-SR3 | Apache Commons Libraries <br/> odl-base-eclipselink-persistence | 2.0.4.v201112161009 | x | odl-base-1.4.5-Helium-SR3 | EclipseLink Persistence API <br/> odl-base-gemini-web | 2.2.0.RELEASE | x | odl-base-1.4.5-Helium-SR3 | Gemini Web <br/> odl-base-tomcat | 7.0.53 | x | odl-base-1.4.5-Helium-SR3 | OpenDaylight Tomcat <br/> odl-base-spring | 3.1.3.RELEASE | x | odl-base-1.4.5-Helium-SR3 | Opendaylight Spring Support <br/> odl-base-spring-web | 3.1.3.RELEASE | x | odl-base-1.4.5-Helium-SR3 | OpenDaylight Spring Web <br/> odl-base-spring-security | 3.1.3.RELEASE | x | odl-base-1.4.5-Helium-SR3 | OpenDaylight Spring Security <br/> odl-mdsal-broker | 1.1.3-Helium-SR3 | x | odl-mdsal-1.1.3-Helium-SR3 | OpenDaylight :: MDSAL :: Broker <br/> odl-config-persister | 0.2.8-Helium-SR3 | x | odl-config-persister-0.2.8-Helium-SR3 | OpenDaylight :: Config Persister <br/> odl-config-startup | 0.2.8-Helium-SR3 | x | odl-config-persister-0.2.8-Helium-SR3 | OpenDaylight :: Config Persister:: Config Startup <br/> odl-protocol-framework | 0.5.3-Helium-SR3 | x | odl-protocol-framework-0.5.3-Helium-SR3 | OpenDaylight :: Protocol Framework <br/> odl-restconf-all | 1.1.3-Helium-SR3 | x | odl-controller-1.1.3-Helium-SR3 | OpenDaylight :: Restconf :: All <br/> odl-restconf | 1.1.3-Helium-SR3 | x | odl-controller-1.1.3-Helium-SR3 | OpenDaylight :: Restconf <br/> odl-restconf-noauth | 1.1.3-Helium-SR3 | x | odl-controller-1.1.3-Helium-SR3 | OpenDaylight :: Restconf <br/> odl-mdsal-apidocs | 1.1.3-Helium-SR3 | x | odl-controller-1.1.3-Helium-SR3 | OpenDaylight :: MDSAL :: APIDOCS <br/> odl-yangtools-models | 0.6.5-Helium-SR3 | x | odl-yangtools-0.6.5-Helium-SR3 | OpenDaylight :: Yangtools :: Models <br/> odl-yangtools-data-binding | 0.6.5-Helium-SR3 | x | odl-yangtools-0.6.5-Helium-SR3 | OpenDaylight :: Yangtools :: Data Binding <br/> odl-yangtools-binding | 0.6.5-Helium-SR3 | x | odl-yangtools-0.6.5-Helium-SR3 | OpenDaylight :: Yangtools :: Binding <br/> odl-yangtools-common | 0.6.5-Helium-SR3 | x | odl-yangtools-0.6.5-Helium-SR3 | OpenDaylight :: Yangtools :: Common <br/> odl-yangtools-binding-generator | 0.6.5-Helium-SR3 | x | odl-yangtools-0.6.5-Helium-SR3 | OpenDaylight :: Yangtools :: Binding Generator <br/> transaction | 1.0.1 | x | enterprise-3.0.1 | OSGi Transaction Manager <br/> odl-netconf-api | 0.2.8-Helium-SR3 | x | odl-netconf-0.2.8-Helium-SR3 | OpenDaylight :: Netconf :: API <br/> odl-netconf-mapping-api | 0.2.8-Helium-SR3 | x | odl-netconf-0.2.8-Helium-SR3 | OpenDaylight :: Netconf :: Mapping API <br/> odl-netconf-util | 0.2.8-Helium-SR3 | x | odl-netconf-0.2.8-Helium-SR3 | <br/> odl-netconf-impl | 0.2.8-Helium-SR3 | x | odl-netconf-0.2.8-Helium-SR3 | OpenDaylight :: Netconf :: Impl <br/> odl-config-netconf-connector | 0.2.8-Helium-SR3 | x | odl-netconf-0.2.8-Helium-SR3 | OpenDaylight :: Netconf :: Connector <br/> odl-netconf-netty-util | 0.2.8-Helium-SR3 | x | odl-netconf-0.2.8-Helium-SR3 | OpenDaylight :: Netconf :: Netty Util <br/> odl-netconf-client | 0.2.8-Helium-SR3 | x | odl-netconf-0.2.8-Helium-SR3 | OpenDaylight :: Netconf :: Client <br/> odl-netconf-monitoring | 0.2.8-Helium-SR3 | x | odl-netconf-0.2.8-Helium-SR3 | OpenDaylight :: Netconf :: Monitoring <br/> pax-jetty | 8.1.14.v20131031 | x | org.ops4j.pax.web-3.1.0 | Provide Jetty engine support <br/> pax-http | 3.1.0 | x | org.ops4j.pax.web-3.1.0 | Implementation of the OSGI HTTP Service <br/> pax-http-whiteboard | 3.1.0 | x | org.ops4j.pax.web-3.1.0 | Provide HTTP Whiteboard pattern support <br/> pax-war | 3.1.0 | x | org.ops4j.pax.web-3.1.0 | Provide support of a full WebContainer</p> <p>The exception is trying to say that config subsystem could not save a service reference named: authz-connector-default because the module its pointing to does not exist. Its pointing to module with type aaa-authz-service and named aaa-authz-service. Looking at attached log (the entire config file is logged), the file does not contain such module and it means the file is incorrect.</p> <p>Looks like an issue with configs from AAA. Will try to reproduce it and ask somebody from AAA about this.</p> <p>Btw. What version of ODL is this ? Is it current master(Lithium) ? <br/> Btw2. AAA features are required by netconf northbound server since it uses AAA to authenticate incoming SSH connections.</p> <p>Just noticed its Helium-SR3 from the comments...</p> <p>Tried with a clean SR3 and could not reproduce. </p> <p>It looks like you are running a distribution that was previously used and reconfigured (because the config file being pushed contains various modules including pcep). Check what is in etc/opendaylight/current and please attach the file.</p> <p>It would also be helpful if you could provide information for what you did with the distribution even in previous runs.</p> <p>In the meantime, deleting etc/opendaylight folder should fix your problem.</p> <p>Yeah I was using the downloaded, pre-built controller from www.opendaylight.org/software/downloads</p> <p>I moved the etc/opendaylight directory elsewhere as you advised, and the problem did not come up during startup. </p> <p>Here's what the opendaylight/etc directory had contained:<br/> etc/opendaylight/karaf/04-xsql.xml<br/> etc/opendaylight/karaf/54-arphandler.xml<br/> etc/opendaylight/karaf/44-nicira-extension.xml<br/> etc/opendaylight/karaf/08-authz-config.xml<br/> etc/opendaylight/karaf/43-msg-spy.xml<br/> etc/opendaylight/karaf/00-netty.xml<br/> etc/opendaylight/karaf/50-packethandler.xml<br/> etc/opendaylight/karaf/33-pcep-segment-routing.xml<br/> etc/opendaylight/karaf/01-mdsal.xml<br/> etc/opendaylight/karaf/39-pcep-provider.xml<br/> etc/opendaylight/karaf/30-statistics-manager.xml<br/> etc/opendaylight/karaf/99-netconf-connector.xml<br/> etc/opendaylight/karaf/32-pcep.xml<br/> etc/opendaylight/karaf/01-netconf.xml<br/> etc/opendaylight/karaf/56-addresstracker.xml<br/> etc/opendaylight/karaf/09-rest-connector.xml<br/> etc/opendaylight/karaf/58-l2switchmain.xml<br/> etc/opendaylight/karaf/50-ovs-nx-api-config.xml<br/> etc/opendaylight/karaf/80-sfc.xml<br/> etc/opendaylight/karaf/69-table-miss-enforcer.xml<br/> etc/opendaylight/karaf/10-rest-connector.xml<br/> etc/opendaylight/karaf/42-openflowplugin.xml<br/> etc/opendaylight/karaf/57-hosttracker.xml<br/> etc/opendaylight/karaf/51-ovs-nx-config.xml<br/> etc/opendaylight/karaf/31-bgp.xml<br/> etc/opendaylight/karaf/80-packetcable.xml<br/> etc/opendaylight/karaf/30-programming.xml<br/> etc/opendaylight/karaf/52-ovs-nx-sal-config.xml<br/> etc/opendaylight/karaf/52-loopremover.xml<br/> etc/opendaylight/karaf/43-openflowjava-nx-api-config.xml<br/> etc/opendaylight/karaf/44-openflowjava-nx-config.xml<br/> etc/opendaylight/karaf/41-bgp-example.xml<br/> etc/opendaylight/karaf/20-tcpmd5.xml<br/> etc/opendaylight/current/controller.currentconfig.xml</p> <p>Helium is no longer supported, is this reproducible in Lithium SR2 or Beryllium?</p> <p>I believe the bug was fixed at some point for one of the Lithium SR versions</p> <p>Thanks</p> Development External issue ID 3034 External issue URL Issue Type Rank 0|i02pj3: