[CONTROLLER-683] Toaster throwing an exception when starting with clustering enabled

Tue, 19 Aug 2014 09:41:33 +0000

To reproduce do the following,
1. build the OpenFlowPlugin distribution.
2. Goto the OpenFlowPlugin distribution directory
3. Rename configuration/initial/05-clustering.xml.conf to configuration/initial/05-clustering.xml
4. Run the controller with the following command,
./run.sh -Xmx4G -XX:MaxPermSize=512m

You will see the following exception show up on startup,

2014-08-12 14:41:55.565 PDT [config-pusher] ERROR o.o.c.c.y.c.t.i.AbstractToasterProviderModule - An error occurred while closing old instance org.opendaylight.controller.config.yang.config.toaster_provider.impl.ToasterProviderModule$1AutoCloseableToaster@6da9a192
java.lang.IllegalStateException: Transaction factory was closed. No further operations allowed.
at com.google.common.base.Preconditions.checkState(Preconditions.java:149) ~[bundlefile:na]
at org.opendaylight.controller.md.sal.dom.broker.impl.AbstractDOMForwardedTransactionFactory.checkNotClosed(AbstractDOMForwardedTransactionFactory.java:214) ~[na:na]
at org.opendaylight.controller.md.sal.dom.broker.impl.AbstractDOMForwardedTransactionFactory.newWriteOnlyTransaction(AbstractDOMForwardedTransactionFactory.java:128) ~[na:na]
at org.opendaylight.controller.sal.dom.broker.osgi.DOMDataBrokerProxy.newWriteOnlyTransaction(DOMDataBrokerProxy.java:33) ~[na:na]
at org.opendaylight.controller.md.sal.binding.impl.ForwardedBindingDataBroker.newWriteOnlyTransaction(ForwardedBindingDataBroker.java:50) ~[na:na]
at org.opendaylight.controller.sample.toaster.provider.OpendaylightToaster.close(OpendaylightToaster.java:103) ~[na:na]
at org.opendaylight.controller.config.yang.config.toaster_provider.impl.ToasterProviderModule$1AutoCloseableToaster.close(ToasterProviderModule.java:79) ~[na:na]
at org.opendaylight.controller.config.yang.config.toaster_provider.impl.AbstractToasterProviderModule.getInstance(AbstractToasterProviderModule.java:97) ~[na:na]
at sun.reflect.GeneratedMethodAccessor156.invoke(Unknown Source) ~[na:na]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_45]
at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_45]
at org.opendaylight.controller.config.manager.impl.dependencyresolver.DependencyResolverManager$1.handleInvocation(DependencyResolverManager.java:152) [bundlefile:na]
at com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:70) [bundlefile:na]
at com.sun.proxy.$Proxy89.getInstance(Unknown Source) [na:na]
at org.opendaylight.controller.config.manager.impl.ConfigTransactionControllerImpl.secondPhaseCommit(ConfigTransactionControllerImpl.java:396) [bundlefile:na]
at org.opendaylight.controller.config.manager.impl.ConfigRegistryImpl.secondPhaseCommit(ConfigRegistryImpl.java:282) [bundlefile:na]
at org.opendaylight.controller.config.manager.impl.ConfigRegistryImpl.commitConfig(ConfigRegistryImpl.java:229) [bundlefile:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_45]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_45]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_45]
at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_45]
at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:75) [na:1.7.0_45]
at sun.reflect.GeneratedMethodAccessor157.invoke(Unknown Source) ~[na:na]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_45]
at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_45]
at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:279) [na:1.7.0_45]
at com.sun.jmx.mbeanserver.ConvertingMethod.invokeWithOpenReturn(ConvertingMethod.java:193) [na:1.7.0_45]
at com.sun.jmx.mbeanserver.ConvertingMethod.invokeWithOpenReturn(ConvertingMethod.java:175) [na:1.7.0_45]
at com.sun.jmx.mbeanserver.MXBeanIntrospector.invokeM2(MXBeanIntrospector.java:117) [na:1.7.0_45]
at com.sun.jmx.mbeanserver.MXBeanIntrospector.invokeM2(MXBeanIntrospector.java:54) [na:1.7.0_45]
at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237) [na:1.7.0_45]
at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138) [na:1.7.0_45]
at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252) [na:1.7.0_45]
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819) [na:1.7.0_45]
at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801) [na:1.7.0_45]
at com.sun.jmx.mbeanserver.MXBeanProxy$InvokeHandler.invoke(MXBeanProxy.java:150) [na:1.7.0_45]
at com.sun.jmx.mbeanserver.MXBeanProxy.invoke(MXBeanProxy.java:167) [na:1.7.0_45]
at javax.management.MBeanServerInvocationHandler.invoke(MBeanServerInvocationHandler.java:252) [na:1.7.0_45]
at com.sun.proxy.$Proxy90.commitConfig(Unknown Source) [na:na]
at org.opendaylight.controller.config.util.ConfigRegistryJMXClient.commitConfig(ConfigRegistryJMXClient.java:102) [bundlefile:na]
at org.opendaylight.controller.netconf.confignetconfconnector.transactions.TransactionProvider.commitTransaction(TransactionProvider.java:110) [bundlefile:na]
at org.opendaylight.controller.netconf.confignetconfconnector.operations.Commit.handleWithNoSubsequentOperations(Commit.java:55) [bundlefile:na]
at org.opendaylight.controller.netconf.util.mapping.AbstractLastNetconfOperation.handle(AbstractLastNetconfOperation.java:33) [bundlefile:na]
at org.opendaylight.controller.netconf.util.mapping.AbstractNetconfOperation.handle(AbstractNetconfOperation.java:105) [bundlefile:na]
at org.opendaylight.controller.netconf.persist.impl.ConfigPusher.sendRequestGetResponseCheckIsOK(ConfigPusher.java:245) [bundlefile:na]
at org.opendaylight.controller.netconf.persist.impl.ConfigPusher.pushConfig(ConfigPusher.java:204) [bundlefile:na]
at org.opendaylight.controller.netconf.persist.impl.ConfigPusher.pushConfigWithConflictingVersionRetries(ConfigPusher.java:93) [bundlefile:na]
at org.opendaylight.controller.netconf.persist.impl.ConfigPusher.pushConfigs(ConfigPusher.java:69) [bundlefile:na]
at org.opendaylight.controller.netconf.persist.impl.osgi.ConfigPersisterActivator$InnerCustomizer$1.run(ConfigPersisterActivator.java:156) [bundlefile:na]
at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]

My initial thought is that this has something to do with the fact that we substitute the in-memory-datastore with the distributed-data-store using a merge configuration (namely by overriding the settings in 01-md-sal.xml with 05-clustering.xml) which results in the Toaster Modules close being invoked. When the toaster tries it's close it fails with this exception.

OpenDaylight JIRA

[CONTROLLER-683] Toaster throwing an exception when starting with clustering enabled