Operating System: All
Platform: All

Attachment file_3034.txt has been added with description: Stack trace of exception

Here's a version of the exception from the attachment with a bunch of the data from the first line trimmed to get it down to something reasonable to git a glance at.

Exception in thread "config-pusher" java.lang.IllegalStateException: Failed to send edit-config for configuration ConfigSnapshot

at org.opendaylight.controller.netconf.persist.impl.ConfigPusherImpl.sendRequestGetResponseCheckIsOK(ConfigPusherImpl.java:293)
at org.opendaylight.controller.netconf.persist.impl.ConfigPusherImpl.pushConfig(ConfigPusherImpl.java:244)
at org.opendaylight.controller.netconf.persist.impl.ConfigPusherImpl.pushConfigWithConflictingVersionRetries(ConfigPusherImpl.java:136)
at org.opendaylight.controller.netconf.persist.impl.ConfigPusherImpl.internalPushConfigs(ConfigPusherImpl.java:109)
at org.opendaylight.controller.netconf.persist.impl.ConfigPusherImpl.process(ConfigPusherImpl.java:76)
at org.opendaylight.controller.netconf.persist.impl.osgi.ConfigPersisterActivator$InnerCustomizer$1.run(ConfigPersisterActivator.java:178)
at java.lang.Thread.run(Thread.java:745)
Caused by: NetconfDocumentedException{message=Unable to save ref name authz-connector-default for instance org.opendaylight.controller:instanceName=aaa-authz-service,TransactionName=ConfigTransaction-28-29,type=Module,moduleFactoryName=aaa-authz-service, errorType=application, errorTag=operation_failed, errorSeverity=error, errorInfo={}}
at org.opendaylight.controller.netconf.confignetconfconnector.operations.editconfig.EditConfig.setServicesOnTransaction(EditConfig.java:164)
at org.opendaylight.controller.netconf.confignetconfconnector.operations.editconfig.EditConfig.test(EditConfig.java:120)
at org.opendaylight.controller.netconf.confignetconfconnector.operations.editconfig.EditConfig.executeTests(EditConfig.java:97)
at org.opendaylight.controller.netconf.confignetconfconnector.operations.editconfig.EditConfig.getResponseInternal(EditConfig.java:76)
at org.opendaylight.controller.netconf.confignetconfconnector.operations.editconfig.EditConfig.handleWithNoSubsequentOperations(EditConfig.java:307)
at org.opendaylight.controller.netconf.util.mapping.AbstractLastNetconfOperation.handle(AbstractLastNetconfOperation.java:33)
at org.opendaylight.controller.netconf.util.mapping.AbstractNetconfOperation.handle(AbstractNetconfOperation.java:105)
at org.opendaylight.controller.netconf.persist.impl.ConfigPusherImpl.sendRequestGetResponseCheckIsOK(ConfigPusherImpl.java:288)
... 6 more

Are you using the aaa-authz service?

I am not sure if that is a tested feature in Helium. Could you try without it?

Also, can you send the output of feature:list -i to give an idea of what features you have installed?

> feature:install odl-base-all
> feature:install odl-netconf-connector-all
> feature:install odl-config-all
> feature:install odl-restconf-all

Does this happen even if odl-netconf-connector-all is installed last?
(I do not really expect this bug to be caused by netconf-connector not getting updated schemas, but it is safer to test for it anyway.)

Vratko - Tried putting odl-netconf-connector-all last, still experienced the exception.

Ed - I had only typed the feature:install commands for the features I listed, but odl-aaa got installed somehow. Is there a way to install those features without getting that pulled in? Should I uninstall the feature and restart the controller to see if it comes back?

Here's the output of feature:list | grep " x ":
odl-aaa-authn | 0.1.3-Helium-SR3 | x | odl-aaa-0.1.3-Helium-SR3 | OpenDaylight :: AAA :: Authentication
odl-aaa-authn-plugin | 0.1.3-Helium-SR3 | x | odl-aaa-0.1.3-Helium-SR3 | OpenDaylight :: AAA :: ODL NETCONF Plugin
standard | 3.0.1 | x | standard-3.0.1 | Karaf standard feature
config | 3.0.1 | x | standard-3.0.1 | Provide OSGi ConfigAdmin support
region | 3.0.1 | x | standard-3.0.1 | Provide Region Support
package | 3.0.1 | x | standard-3.0.1 | Package commands and mbeans
http | 3.0.1 | x | standard-3.0.1 | Implementation of the OSGI HTTP Service
war | 3.0.1 | x | standard-3.0.1 | Turn Karaf as a full WebContainer
kar | 3.0.1 | x | standard-3.0.1 | Provide KAR (KARaf archive) support
ssh | 3.0.1 | x | standard-3.0.1 | Provide a SSHd server on Karaf
management | 3.0.1 | x | standard-3.0.1 | Provide a JMX MBeanServer and a set of MBeans in K
odl-netconf-connector-all | 1.1.3-Helium-SR3 | x | odl-controller-1.1.3-Helium-SR3 | OpenDaylight :: Netconf Connector :: All
odl-netconf-connector | 1.1.3-Helium-SR3 | x | odl-controller-1.1.3-Helium-SR3 | OpenDaylight :: Netconf Connector :: Netconf Conne
odl-netconf-connector-ssh | 1.1.3-Helium-SR3 | x | odl-controller-1.1.3-Helium-SR3 | OpenDaylight :: Netconf Connector :: Netconf Conne
odl-netconf-ssh | 0.2.8-Helium-SR3 | x | odl-controller-1.1.3-Helium-SR3 | OpenDaylight :: Netconf Connector :: SSH
odl-netconf-tcp | 0.2.8-Helium-SR3 | x | odl-controller-1.1.3-Helium-SR3 | OpenDaylight :: Netconf Connector :: TCP
odl-config-all | 0.2.8-Helium-SR3 | x | odl-config-0.2.8-Helium-SR3 | OpenDaylight :: Config :: All
odl-mdsal-common | 1.1.3-Helium-SR3 | x | odl-config-0.2.8-Helium-SR3 | OpenDaylight :: Config :: All
odl-config-api | 0.2.8-Helium-SR3 | x | odl-config-0.2.8-Helium-SR3 | OpenDaylight :: Config :: API
odl-config-netty-config-api | 0.2.8-Helium-SR3 | x | odl-config-0.2.8-Helium-SR3 | OpenDaylight :: Config :: Netty Config API
odl-config-core | 0.2.8-Helium-SR3 | x | odl-config-0.2.8-Helium-SR3 | OpenDaylight :: Config :: Core
odl-config-manager | 0.2.8-Helium-SR3 | x | odl-config-0.2.8-Helium-SR3 | OpenDaylight :: Config :: Manager
odl-config-netty | 0.2.8-Helium-SR3 | x | odl-config-persister-0.2.8-Helium-SR3 | OpenDaylight :: Config-Netty
odl-base-all | 1.4.5-Helium-SR3 | x | odl-base-1.4.5-Helium-SR3 | OpenDaylight Controller
odl-base-dummy-console | 1.1.3-Helium-SR3 | x | odl-base-1.4.5-Helium-SR3 | Temporary Dummy Console
odl-base-felix-dm | 3.1.0 | x | odl-base-1.4.5-Helium-SR3 | Felix Dependency Manager
odl-base-aries-spi-fly | 1.0.0 | x | odl-base-1.4.5-Helium-SR3 | Aries SPI Fly
odl-base-netty | 4.0.23.Final | x | odl-base-1.4.5-Helium-SR3 |
odl-base-jersey | 1.17 | x | odl-base-1.4.5-Helium-SR3 | Jersey
odl-base-jackson | 2.3.2 | x | odl-base-1.4.5-Helium-SR3 | Jackson JAX-RS
odl-base-slf4j | 1.7.2 | x | odl-base-1.4.5-Helium-SR3 | SLF4J Logging
odl-base-apache-commons | 1.4.5-Helium-SR3 | x | odl-base-1.4.5-Helium-SR3 | Apache Commons Libraries
odl-base-eclipselink-persistence | 2.0.4.v201112161009 | x | odl-base-1.4.5-Helium-SR3 | EclipseLink Persistence API
odl-base-gemini-web | 2.2.0.RELEASE | x | odl-base-1.4.5-Helium-SR3 | Gemini Web
odl-base-tomcat | 7.0.53 | x | odl-base-1.4.5-Helium-SR3 | OpenDaylight Tomcat
odl-base-spring | 3.1.3.RELEASE | x | odl-base-1.4.5-Helium-SR3 | Opendaylight Spring Support
odl-base-spring-web | 3.1.3.RELEASE | x | odl-base-1.4.5-Helium-SR3 | OpenDaylight Spring Web
odl-base-spring-security | 3.1.3.RELEASE | x | odl-base-1.4.5-Helium-SR3 | OpenDaylight Spring Security
odl-mdsal-broker | 1.1.3-Helium-SR3 | x | odl-mdsal-1.1.3-Helium-SR3 | OpenDaylight :: MDSAL :: Broker
odl-config-persister | 0.2.8-Helium-SR3 | x | odl-config-persister-0.2.8-Helium-SR3 | OpenDaylight :: Config Persister
odl-config-startup | 0.2.8-Helium-SR3 | x | odl-config-persister-0.2.8-Helium-SR3 | OpenDaylight :: Config Persister:: Config Startup
odl-protocol-framework | 0.5.3-Helium-SR3 | x | odl-protocol-framework-0.5.3-Helium-SR3 | OpenDaylight :: Protocol Framework
odl-restconf-all | 1.1.3-Helium-SR3 | x | odl-controller-1.1.3-Helium-SR3 | OpenDaylight :: Restconf :: All
odl-restconf | 1.1.3-Helium-SR3 | x | odl-controller-1.1.3-Helium-SR3 | OpenDaylight :: Restconf
odl-restconf-noauth | 1.1.3-Helium-SR3 | x | odl-controller-1.1.3-Helium-SR3 | OpenDaylight :: Restconf
odl-mdsal-apidocs | 1.1.3-Helium-SR3 | x | odl-controller-1.1.3-Helium-SR3 | OpenDaylight :: MDSAL :: APIDOCS
odl-yangtools-models | 0.6.5-Helium-SR3 | x | odl-yangtools-0.6.5-Helium-SR3 | OpenDaylight :: Yangtools :: Models
odl-yangtools-data-binding | 0.6.5-Helium-SR3 | x | odl-yangtools-0.6.5-Helium-SR3 | OpenDaylight :: Yangtools :: Data Binding
odl-yangtools-binding | 0.6.5-Helium-SR3 | x | odl-yangtools-0.6.5-Helium-SR3 | OpenDaylight :: Yangtools :: Binding
odl-yangtools-common | 0.6.5-Helium-SR3 | x | odl-yangtools-0.6.5-Helium-SR3 | OpenDaylight :: Yangtools :: Common
odl-yangtools-binding-generator | 0.6.5-Helium-SR3 | x | odl-yangtools-0.6.5-Helium-SR3 | OpenDaylight :: Yangtools :: Binding Generator
transaction | 1.0.1 | x | enterprise-3.0.1 | OSGi Transaction Manager
odl-netconf-api | 0.2.8-Helium-SR3 | x | odl-netconf-0.2.8-Helium-SR3 | OpenDaylight :: Netconf :: API
odl-netconf-mapping-api | 0.2.8-Helium-SR3 | x | odl-netconf-0.2.8-Helium-SR3 | OpenDaylight :: Netconf :: Mapping API
odl-netconf-util | 0.2.8-Helium-SR3 | x | odl-netconf-0.2.8-Helium-SR3 |
odl-netconf-impl | 0.2.8-Helium-SR3 | x | odl-netconf-0.2.8-Helium-SR3 | OpenDaylight :: Netconf :: Impl
odl-config-netconf-connector | 0.2.8-Helium-SR3 | x | odl-netconf-0.2.8-Helium-SR3 | OpenDaylight :: Netconf :: Connector
odl-netconf-netty-util | 0.2.8-Helium-SR3 | x | odl-netconf-0.2.8-Helium-SR3 | OpenDaylight :: Netconf :: Netty Util
odl-netconf-client | 0.2.8-Helium-SR3 | x | odl-netconf-0.2.8-Helium-SR3 | OpenDaylight :: Netconf :: Client
odl-netconf-monitoring | 0.2.8-Helium-SR3 | x | odl-netconf-0.2.8-Helium-SR3 | OpenDaylight :: Netconf :: Monitoring
pax-jetty | 8.1.14.v20131031 | x | org.ops4j.pax.web-3.1.0 | Provide Jetty engine support
pax-http | 3.1.0 | x | org.ops4j.pax.web-3.1.0 | Implementation of the OSGI HTTP Service
pax-http-whiteboard | 3.1.0 | x | org.ops4j.pax.web-3.1.0 | Provide HTTP Whiteboard pattern support
pax-war | 3.1.0 | x | org.ops4j.pax.web-3.1.0 | Provide support of a full WebContainer

The exception is trying to say that config subsystem could not save a service reference named: authz-connector-default because the module its pointing to does not exist. Its pointing to module with type aaa-authz-service and named aaa-authz-service. Looking at attached log (the entire config file is logged), the file does not contain such module and it means the file is incorrect.

Looks like an issue with configs from AAA. Will try to reproduce it and ask somebody from AAA about this.

Btw. What version of ODL is this ? Is it current master(Lithium) ?
Btw2. AAA features are required by netconf northbound server since it uses AAA to authenticate incoming SSH connections.

Just noticed its Helium-SR3 from the comments...

Tried with a clean SR3 and could not reproduce.

It looks like you are running a distribution that was previously used and reconfigured (because the config file being pushed contains various modules including pcep). Check what is in etc/opendaylight/current and please attach the file.

It would also be helpful if you could provide information for what you did with the distribution even in previous runs.

In the meantime, deleting etc/opendaylight folder should fix your problem.

Yeah I was using the downloaded, pre-built controller from www.opendaylight.org/software/downloads

I moved the etc/opendaylight directory elsewhere as you advised, and the problem did not come up during startup.

Here's what the opendaylight/etc directory had contained:
etc/opendaylight/karaf/04-xsql.xml
etc/opendaylight/karaf/54-arphandler.xml
etc/opendaylight/karaf/44-nicira-extension.xml
etc/opendaylight/karaf/08-authz-config.xml
etc/opendaylight/karaf/43-msg-spy.xml
etc/opendaylight/karaf/00-netty.xml
etc/opendaylight/karaf/50-packethandler.xml
etc/opendaylight/karaf/33-pcep-segment-routing.xml
etc/opendaylight/karaf/01-mdsal.xml
etc/opendaylight/karaf/39-pcep-provider.xml
etc/opendaylight/karaf/30-statistics-manager.xml
etc/opendaylight/karaf/99-netconf-connector.xml
etc/opendaylight/karaf/32-pcep.xml
etc/opendaylight/karaf/01-netconf.xml
etc/opendaylight/karaf/56-addresstracker.xml
etc/opendaylight/karaf/09-rest-connector.xml
etc/opendaylight/karaf/58-l2switchmain.xml
etc/opendaylight/karaf/50-ovs-nx-api-config.xml
etc/opendaylight/karaf/80-sfc.xml
etc/opendaylight/karaf/69-table-miss-enforcer.xml
etc/opendaylight/karaf/10-rest-connector.xml
etc/opendaylight/karaf/42-openflowplugin.xml
etc/opendaylight/karaf/57-hosttracker.xml
etc/opendaylight/karaf/51-ovs-nx-config.xml
etc/opendaylight/karaf/31-bgp.xml
etc/opendaylight/karaf/80-packetcable.xml
etc/opendaylight/karaf/30-programming.xml
etc/opendaylight/karaf/52-ovs-nx-sal-config.xml
etc/opendaylight/karaf/52-loopremover.xml
etc/opendaylight/karaf/43-openflowjava-nx-api-config.xml
etc/opendaylight/karaf/44-openflowjava-nx-config.xml
etc/opendaylight/karaf/41-bgp-example.xml
etc/opendaylight/karaf/20-tcpmd5.xml
etc/opendaylight/current/controller.currentconfig.xml

Helium is no longer supported, is this reproducible in Lithium SR2 or Beryllium?

I believe the bug was fixed at some point for one of the Lithium SR versions

Thanks