[USC-12] USC features seem to have issues with 3-node cluster environment

Tue, 22 May 2018 20:27:41 +0000

These exception are observed in stable/lithium when bringing usc in 3-node cluster environment:

https://jenkins.opendaylight.org/releng/view/integration/job/integration-csit-3node-basic-all-stable-lithium/457/console

015-10-29 10:21:38,589 | INFO | lt-dispatcher-18 | rovider$RemoteDeadLetterActorRef | 144 - com.typesafe.akka.slf4j - 2.3.10 | Message [akka.cluster.InternalClusterAction$InitJoinNack] from Actorakka://odl-cluster-rpc/system/cluster/core/daemon#-643930024 to Actor[akka://odl-cluster-rpc/deadLetters] was not delivered. [10] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
2015-10-29 10:21:39,033 | INFO | lt-dispatcher-16 | Cluster(akka://odl-cluster-usc) | 144 - com.typesafe.akka.slf4j - 2.3.10 | Cluster Node [akka.tcp://odl-cluster-usc@127.0.0.1:2552] - Leader is moving node [akka.tcp://odl-cluster-usc@127.0.0.1:2552] to [Up]
2015-10-29 10:21:39,034 | INFO | ult-dispatcher-4 | UscCommunicatorActor | 505 - org.opendaylight.usc.channel-impl - 1.0.3.SNAPSHOT | handleReceive received message:MemberUp(Member(address = akka.tcp://odl-cluster-usc@127.0.0.1:2552, status = Up))
2015-10-29 10:21:39,028 | ERROR | config-pusher | ConfigTransactionControllerImpl | 92 - org.opendaylight.controller.config-manager - 0.3.3.SNAPSHOT | Commit failed on ModuleIdentifier

{factoryName='instruction-scheduler-impl', instanceName='global-instruction-scheduler'} in transaction TransactionIdentifier{name='ConfigTransaction-189-191'}
java.lang.IllegalStateException: Conflicting instruction queue found
at com.google.common.base.Preconditions.checkState(Preconditions.java:173)
at org.opendaylight.bgpcep.programming.impl.ProgrammingServiceImpl.<init>(ProgrammingServiceImpl.java:123)
at org.opendaylight.controller.config.yang.programming.impl.InstructionSchedulerImplModule.createInstance(InstructionSchedulerImplModule.java:57)
at org.opendaylight.controller.config.spi.AbstractModule.getInstance(AbstractModule.java:73)
at sun.reflect.GeneratedMethodAccessor27.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[:1.7.0_85]
at java.lang.reflect.Method.invoke(Method.java:606)[:1.7.0_85]
at org.opendaylight.controller.config.manager.impl.dependencyresolver.DependencyResolverManager$ModuleInvocationHandler.handleInvocation(DependencyResolverManager.java:150)
at com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:87)
at com.sun.proxy.$Proxy98.getInstance(Unknown Source)
at org.opendaylight.controller.config.manager.impl.ConfigTransactionControllerImpl.secondPhaseCommit(ConfigTransactionControllerImpl.java:399)
at org.opendaylight.controller.config.manager.impl.ConfigRegistryImpl.secondPhaseCommit(ConfigRegistryImpl.java:280)
at org.opendaylight.controller.config.manager.impl.ConfigRegistryImpl.commitConfig(ConfigRegistryImpl.java:227)
at sun.reflect.GeneratedMethodAccessor226.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[:1.7.0_85]
at java.lang.reflect.Method.invoke(Method.java:606)[:1.7.0_85]
at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:75)[:1.7.0_85]
at sun.reflect.GeneratedMethodAccessor173.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[:1.7.0_85]
at java.lang.reflect.Method.invoke(Method.java:606)[:1.7.0_85]
at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:279)[:1.7.0_85]
at com.sun.jmx.mbeanserver.ConvertingMethod.invokeWithOpenReturn(ConvertingMethod.java:193)[:1.7.0_85]
at com.sun.jmx.mbeanserver.ConvertingMethod.invokeWithOpenReturn(ConvertingMethod.java:175)[:1.7.0_85]
at com.sun.jmx.mbeanserver.MXBeanIntrospector.invokeM2(MXBeanIntrospector.java:117)[:1.7.0_85]
at com.sun.jmx.mbeanserver.MXBeanIntrospector.invokeM2(MXBeanIntrospector.java:54)[:1.7.0_85]
at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)[:1.7.0_85]
at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)[:1.7.0_85]
at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252)[:1.7.0_85]
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)[:1.7.0_85]
at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)[:1.7.0_85]
at com.sun.jmx.mbeanserver.MXBeanProxy$InvokeHandler.invoke(MXBeanProxy.java:150)[:1.7.0_85]
at com.sun.jmx.mbeanserver.MXBeanProxy.invoke(MXBeanProxy.java:167)[:1.7.0_85]
at javax.management.MBeanServerInvocationHandler.invoke(MBeanServerInvocationHandler.java:258)[:1.7.0_85]
at com.sun.proxy.$Proxy16.commitConfig(Unknown Source)[73:org.opendaylight.controller.config-api:0.3.3.SNAPSHOT]
at org.opendaylight.controller.config.util.ConfigRegistryJMXClient.commitConfig(ConfigRegistryJMXClient.java:102)[87:org.opendaylight.controller.config-util:0.3.3.SNAPSHOT]
at org.opendaylight.controller.netconf.confignetconfconnector.transactions.TransactionProvider.commitTransaction(TransactionProvider.java:138)[107:org.opendaylight.controller.config-netconf-connector:0.3.3.SNAPSHOT]
at org.opendaylight.controller.netconf.confignetconfconnector.operations.Commit.handleWithNoSubsequentOperations(Commit.java:54)[107:org.opendaylight.controller.config-netconf-connector:0.3.3.SNAPSHOT]
at org.opendaylight.controller.netconf.util.mapping.AbstractLastNetconfOperation.handle(AbstractLastNetconfOperation.java:33)[105:org.opendaylight.controller.netconf-util:0.3.3.SNAPSHOT]
at org.opendaylight.controller.netconf.util.mapping.AbstractNetconfOperation.handle(AbstractNetconfOperation.java:100)[105:org.opendaylight.controller.netconf-util:0.3.3.SNAPSHOT]
at org.opendaylight.controller.netconf.persist.impl.ConfigPusherImpl.sendRequestGetResponseCheckIsOK(ConfigPusherImpl.java:342)[110:org.opendaylight.controller.config-persister-impl:0.3.3.SNAPSHOT]
at org.opendaylight.controller.netconf.persist.impl.ConfigPusherImpl.pushConfig(ConfigPusherImpl.java:296)[110:org.opendaylight.controller.config-persister-impl:0.3.3.SNAPSHOT]
at org.opendaylight.controller.netconf.persist.impl.ConfigPusherImpl.pushConfigWithConflictingVersionRetries(ConfigPusherImpl.java:135)[110:org.opendaylight.controller.config-persister-impl:0.3.3.SNAPSHOT]
at org.opendaylight.controller.netconf.persist.impl.ConfigPusherImpl.internalPushConfigs(ConfigPusherImpl.java:103)[110:org.opendaylight.controller.config-persister-impl:0.3.3.SNAPSHOT]
at org.opendaylight.controller.netconf.persist.impl.ConfigPusherImpl.process(ConfigPusherImpl.java:76)[110:org.opendaylight.controller.config-persister-impl:0.3.3.SNAPSHOT]
at org.opendaylight.controller.netconf.persist.impl.osgi.ConfigPersisterActivator$InnerCustomizer$1.run(ConfigPersisterActivator.java:181)[110:org.opendaylight.controller.config-persister-impl:0.3.3.SNAPSHOT]
at java.lang.Thread.run(Thread.java:745)[:1.7.0_85]
2015-10-29 10:21:39,037 | ERROR | config-pusher | ConfigRegistryImpl | 92 - org.opendaylight.controller.config-manager - 0.3.3.SNAPSHOT | Configuration Transaction failed on 2PC, server is unhealthy
java.lang.IllegalStateException: Error - getInstance() failed for ModuleIdentifier{factoryName='instruction-scheduler-impl', instanceName='global-instruction-scheduler'}

in transaction TransactionIdentifier

{name='ConfigTransaction-189-191'}

at org.opendaylight.controller.config.manager.impl.ConfigTransactionControllerImpl.secondPhaseCommit(ConfigTransactionControllerImpl.java:405)
at org.opendaylight.controller.config.manager.impl.ConfigRegistryImpl.secondPhaseCommit(ConfigRegistryImpl.java:280)
at org.opendaylight.controller.config.manager.impl.ConfigRegistryImpl.commitConfig(ConfigRegistryImpl.java:227)
at sun.reflect.GeneratedMethodAccessor226.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[:1.7.0_85]
at java.lang.reflect.Method.invoke(Method.java:606)[:1.7.0_85]
at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:75)[:1.7.0_85]
at sun.reflect.GeneratedMethodAccessor173.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[:1.7.0_85]
at java.lang.reflect.Method.invoke(Method.java:606)[:1.7.0_85]
at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:279)[:1.7.0_85]
at com.sun.jmx.mbeanserver.ConvertingMethod.invokeWithOpenReturn(ConvertingMethod.java:193)[:1.7.0_85]
at com.sun.jmx.mbeanserver.ConvertingMethod.invokeWithOpenReturn(ConvertingMethod.java:175)[:1.7.0_85]
at com.sun.jmx.mbeanserver.MXBeanIntrospector.invokeM2(MXBeanIntrospector.java:117)[:1.7.0_85]
at com.sun.jmx.mbeanserver.MXBeanIntrospector.invokeM2(MXBeanIntrospector.java:54)[:1.7.0_85]
at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)[:1.7.0_85]
at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)[:1.7.0_85]
at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252)[:1.7.0_85]
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)[:1.7.0_85]
at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)[:1.7.0_85]
at com.sun.jmx.mbeanserver.MXBeanProxy$InvokeHandler.invoke(MXBeanProxy.java:150)[:1.7.0_85]
at com.sun.jmx.mbeanserver.MXBeanProxy.invoke(MXBeanProxy.java:167)[:1.7.0_85]
at javax.management.MBeanServerInvocationHandler.invoke(MBeanServerInvocationHandler.java:258)[:1.7.0_85]
at com.sun.proxy.$Proxy16.commitConfig(Unknown Source)[73:org.opendaylight.controller.config-api:0.3.3.SNAPSHOT]
at org.opendaylight.controller.config.util.ConfigRegistryJMXClient.commitConfig(ConfigRegistryJMXClient.java:102)[87:org.opendaylight.controller.config-util:0.3.3.SNAPSHOT]
at org.opendaylight.controller.netconf.confignetconfconnector.transactions.TransactionProvider.commitTransaction(TransactionProvider.java:138)[107:org.opendaylight.controller.config-netconf-connector:0.3.3.SNAPSHOT]
at org.opendaylight.controller.netconf.confignetconfconnector.operations.Commit.handleWithNoSubsequentOperations(Commit.java:54)[107:org.opendaylight.controller.config-netconf-connector:0.3.3.SNAPSHOT]
at org.opendaylight.controller.netconf.util.mapping.AbstractLastNetconfOperation.handle(AbstractLastNetconfOperation.java:33)[105:org.opendaylight.controller.netconf-util:0.3.3.SNAPSHOT]
at org.opendaylight.controller.netconf.util.mapping.AbstractNetconfOperation.handle(AbstractNetconfOperation.java:100)[105:org.opendaylight.controller.netconf-util:0.3.3.SNAPSHOT]
at org.opendaylight.controller.netconf.persist.impl.ConfigPusherImpl.sendRequestGetResponseCheckIsOK(ConfigPusherImpl.java:342)[110:org.opendaylight.controller.config-persister-impl:0.3.3.SNAPSHOT]
at org.opendaylight.controller.netconf.persist.impl.ConfigPusherImpl.pushConfig(ConfigPusherImpl.java:296)[110:org.opendaylight.controller.config-persister-impl:0.3.3.SNAPSHOT]
at org.opendaylight.controller.netconf.persist.impl.ConfigPusherImpl.pushConfigWithConflictingVersionRetries(ConfigPusherImpl.java:135)[110:org.opendaylight.controller.config-persister-impl:0.3.3.SNAPSHOT]
at org.opendaylight.controller.netconf.persist.impl.ConfigPusherImpl.internalPushConfigs(ConfigPusherImpl.java:103)[110:org.opendaylight.controller.config-persister-impl:0.3.3.SNAPSHOT]
at org.opendaylight.controller.netconf.persist.impl.ConfigPusherImpl.process(ConfigPusherImpl.java:76)[110:org.opendaylight.controller.config-persister-impl:0.3.3.SNAPSHOT]
at org.opendaylight.controller.netconf.persist.impl.osgi.ConfigPersisterActivator$InnerCustomizer$1.run(ConfigPersisterActivator.java:181)[110:org.opendaylight.controller.config-persister-impl:0.3.3.SNAPSHOT]
at java.lang.Thread.run(Thread.java:745)[:1.7.0_85]
Caused by: java.lang.IllegalStateException: Conflicting instruction queue found
at com.google.common.base.Preconditions.checkState(Preconditions.java:173)
at org.opendaylight.bgpcep.programming.impl.ProgrammingServiceImpl.<init>(ProgrammingServiceImpl.java:123)
at org.opendaylight.controller.config.yang.programming.impl.InstructionSchedulerImplModule.createInstance(InstructionSchedulerImplModule.java:57)
at org.opendaylight.controller.config.spi.AbstractModule.getInstance(AbstractModule.java:73)
at sun.reflect.GeneratedMethodAccessor27.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[:1.7.0_85]
at java.lang.reflect.Method.invoke(Method.java:606)[:1.7.0_85]
at org.opendaylight.controller.config.manager.impl.dependencyresolver.DependencyResolverManager$ModuleInvocationHandler.handleInvocation(DependencyResolverManager.java:150)
at com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:87)
at com.sun.proxy.$Proxy98.getInstance(Unknown Source)
at org.opendaylight.controller.config.manager.impl.ConfigTransactionControllerImpl.secondPhaseCommit(ConfigTransactionControllerImpl.java:399)
... 35 more
2015-10-29 10:21:39,126 | INFO | lt-dispatcher-16 | rovider$RemoteDeadLetterActorRef | 144 - com.typesafe.akka.slf4j - 2.3.10 | Message [akka.cluster.InternalClusterAction$InitJoin$] from Actorakka://odl-cluster-rpc/system/cluster/core/daemon/joinSeedNodeProcess-1#1073483844 to Actor[akka://odl-cluster-rpc/deadLetters] was not delivered. [11] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.

OpenDaylight JIRA

[USC-12] USC features seem to have issues with 3-node cluster environment