[OPNFLWPLUG-849] "Initial topology export failed" while pushing 42-openflowplugin-Li.xml

Mon, 27 Sep 2021 09:01:59 +0000

This is a specific symptom visible in karaf log which may explain why ~~CONTROLLER-1550~~ (or similar failure) happens in 3node jobs.

The recent Carbon snapshot builds usually succeed in installing flows, but may fail to verify [0] the flows are present on operational datastore.

The underlying issue is that topology-manager code is not ready for "ConflictingModificationAppliedException: Node was created by other transaction."
This is similar to ~~BGPCEP-596~~ and ~~NETCONF-346~~ but in topology-manager it does not lead to "Server is unhealthy". "continuing anyway" strategy is not enough to make openflow work though.

Relevant segment of full [1] karaf.log:

2017-02-07 05:22:39,780 | INFO | config-pusher | ConfigPusherImpl | 150 - org.opendaylight.controller.config-persister-impl - 0.6.0.SNAPSHOT | Pushing configuration snapshot 42-openflowplugin-Li.xml(odl-openflowplugin-app-config-pusher,odl-openflowplugin-app-config-pusher)
2017-02-07 05:22:39,808 | WARN | lt-dispatcher-33 | ConcurrentDOMDataBroker | 230 - org.opendaylight.controller.sal-distributed-datastore - 1.5.0.SNAPSHOT | Tx: DOM-7 Error during phase CAN_COMMIT, starting Abort
OptimisticLockFailedException

{message=Optimistic lock failed., errorList=[RpcError [message=Optimistic lock failed., severity=ERROR, errorType=APPLICATION, tag=resource-denied, applicationTag=null, info=null, cause=org.opendaylight.yangtools.yang.data.api.schema.tree.ConflictingModificationAppliedException: Node was created by other transaction.]]}

at org.opendaylight.controller.cluster.datastore.ShardDataTree.lambda$processNextPendingTransaction$0(ShardDataTree.java:578)
at org.opendaylight.controller.cluster.datastore.ShardDataTree.processNextPending(ShardDataTree.java:615)
at org.opendaylight.controller.cluster.datastore.ShardDataTree.processNextPendingTransaction(ShardDataTree.java:563)
at org.opendaylight.controller.cluster.datastore.ShardDataTree.startCanCommit(ShardDataTree.java:641)
at org.opendaylight.controller.cluster.datastore.SimpleShardDataTreeCohort.canCommit(SimpleShardDataTreeCohort.java:76)
at org.opendaylight.controller.cluster.datastore.CohortEntry.canCommit(CohortEntry.java:97)
at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator.handleCanCommit(ShardCommitCoordinator.java:236)
at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator.handleForwardedReadyTransaction(ShardCommitCoordinator.java:114)
at org.opendaylight.controller.cluster.datastore.Shard.handleForwardedReadyTransaction(Shard.java:594)
at org.opendaylight.controller.cluster.datastore.Shard.handleNonRaftCommand(Shard.java:291)
at org.opendaylight.controller.cluster.raft.RaftActor.handleCommand(RaftActor.java:268)
at org.opendaylight.controller.cluster.common.actor.AbstractUntypedPersistentActor.onReceiveCommand(AbstractUntypedPersistentActor.java:31)
at akka.persistence.UntypedPersistentActor.onReceive(PersistentActor.scala:170)
at org.opendaylight.controller.cluster.common.actor.MeteringBehavior.apply(MeteringBehavior.java:104)
at akka.actor.ActorCell$$anonfun$become$1.applyOrElse(ActorCell.scala:544)
at akka.actor.Actor$class.aroundReceive(Actor.scala:496)
at akka.persistence.UntypedPersistentActor.akka$persistence$Eventsourced$$super$aroundReceive(PersistentActor.scala:168)
at akka.persistence.Eventsourced$$anon$1.stateReceive(Eventsourced.scala:664)
at akka.persistence.Eventsourced$class.aroundReceive(Eventsourced.scala:183)
at akka.persistence.UntypedPersistentActor.aroundReceive(PersistentActor.scala:168)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
at akka.actor.ActorCell.invoke(ActorCell.scala:495)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
at akka.dispatch.Mailbox.run(Mailbox.scala:224)
at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: org.opendaylight.yangtools.yang.data.api.schema.tree.ConflictingModificationAppliedException: Node was created by other transaction.
at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkWriteApplicable(SchemaAwareApplyOperation.java:174)
at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:128)
at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkChildPreconditions(AbstractNodeContainerModificationStrategy.java:305)
at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkMergeApplicable(AbstractNodeContainerModificationStrategy.java:313)
at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:131)
at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkChildPreconditions(AbstractNodeContainerModificationStrategy.java:305)
at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkMergeApplicable(AbstractNodeContainerModificationStrategy.java:313)
at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:131)
at org.opendaylight.yangtools.yang.data.impl.schema.tree.StructuralContainerModificationStrategy.checkApplicable(StructuralContainerModificationStrategy.java:99)
at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkChildPreconditions(AbstractNodeContainerModificationStrategy.java:305)
at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkTouchApplicable(AbstractNodeContainerModificationStrategy.java:288)
at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:125)
at org.opendaylight.yangtools.yang.data.impl.schema.tree.RootModificationApplyOperation.checkApplicable(RootModificationApplyOperation.java:72)
at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractDataTreeTip.validate(AbstractDataTreeTip.java:35)
at org.opendaylight.controller.cluster.datastore.ShardDataTree.lambda$processNextPendingTransaction$0(ShardDataTree.java:570)
... 28 more
2017-02-07 05:22:39,826 | WARN | rint Extender: 2 | FlowCapableTopologyProvider | 253 - org.opendaylight.openflowplugin.applications.topology-manager - 0.4.0.SNAPSHOT | Initial topology export failed, continuing anyway
java.util.concurrent.ExecutionException: OptimisticLockFailedException

at org.opendaylight.yangtools.util.concurrent.MappingCheckedFuture.wrapInExecutionException(MappingCheckedFuture.java:64)
at org.opendaylight.yangtools.util.concurrent.MappingCheckedFuture.get(MappingCheckedFuture.java:77)
at org.opendaylight.openflowplugin.applications.topology.manager.FlowCapableTopologyProvider.start(FlowCapableTopologyProvider.java:61)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.aries.blueprint.utils.ReflectionUtils.invoke(ReflectionUtils.java:299)
at org.apache.aries.blueprint.container.BeanRecipe.invoke(BeanRecipe.java:980)
at org.apache.aries.blueprint.container.BeanRecipe.runBeanProcInit(BeanRecipe.java:736)
at org.apache.aries.blueprint.container.BeanRecipe.internalCreate2(BeanRecipe.java:848)
at org.apache.aries.blueprint.container.BeanRecipe.internalCreate(BeanRecipe.java:811)
at org.apache.aries.blueprint.di.AbstractRecipe$1.call(AbstractRecipe.java:79)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.apache.aries.blueprint.di.AbstractRecipe.create(AbstractRecipe.java:88)
at org.apache.aries.blueprint.container.BlueprintRepository.createInstances(BlueprintRepository.java:255)
at org.apache.aries.blueprint.container.BlueprintRepository.createAll(BlueprintRepository.java:186)
at org.apache.aries.blueprint.container.BlueprintContainerImpl.instantiateEagerComponents(BlueprintContainerImpl.java:724)
at org.apache.aries.blueprint.container.BlueprintContainerImpl.doRun(BlueprintContainerImpl.java:411)
at org.apache.aries.blueprint.container.BlueprintContainerImpl.run(BlueprintContainerImpl.java:276)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.apache.aries.blueprint.container.ExecutorServiceWrapper.run(ExecutorServiceWrapper.java:106)
at org.apache.aries.blueprint.utils.threading.impl.DiscardableRunnable.run(DiscardableRunnable.java:48)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: OptimisticLockFailedException

at org.opendaylight.controller.cluster.datastore.ShardDataTree.lambda$processNextPendingTransaction$0(ShardDataTree.java:578)
at org.opendaylight.controller.cluster.datastore.ShardDataTree.processNextPending(ShardDataTree.java:615)
at org.opendaylight.controller.cluster.datastore.ShardDataTree.processNextPendingTransaction(ShardDataTree.java:563)
at org.opendaylight.controller.cluster.datastore.ShardDataTree.startCanCommit(ShardDataTree.java:641)
at org.opendaylight.controller.cluster.datastore.SimpleShardDataTreeCohort.canCommit(SimpleShardDataTreeCohort.java:76)
at org.opendaylight.controller.cluster.datastore.CohortEntry.canCommit(CohortEntry.java:97)
at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator.handleCanCommit(ShardCommitCoordinator.java:236)
at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator.handleForwardedReadyTransaction(ShardCommitCoordinator.java:114)
at org.opendaylight.controller.cluster.datastore.Shard.handleForwardedReadyTransaction(Shard.java:594)
at org.opendaylight.controller.cluster.datastore.Shard.handleNonRaftCommand(Shard.java:291)
at org.opendaylight.controller.cluster.raft.RaftActor.handleCommand(RaftActor.java:268)
at org.opendaylight.controller.cluster.common.actor.AbstractUntypedPersistentActor.onReceiveCommand(AbstractUntypedPersistentActor.java:31)
at akka.persistence.UntypedPersistentActor.onReceive(PersistentActor.scala:170)
at org.opendaylight.controller.cluster.common.actor.MeteringBehavior.apply(MeteringBehavior.java:104)
at akka.actor.ActorCell$$anonfun$become$1.applyOrElse(ActorCell.scala:544)
at akka.actor.Actor$class.aroundReceive(Actor.scala:496)
at akka.persistence.UntypedPersistentActor.akka$persistence$Eventsourced$$super$aroundReceive(PersistentActor.scala:168)
at akka.persistence.Eventsourced$$anon$1.stateReceive(Eventsourced.scala:664)
at akka.persistence.Eventsourced$class.aroundReceive(Eventsourced.scala:183)
at akka.persistence.UntypedPersistentActor.aroundReceive(PersistentActor.scala:168)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
at akka.actor.ActorCell.invoke(ActorCell.scala:495)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
at akka.dispatch.Mailbox.run(Mailbox.scala:224)
at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: org.opendaylight.yangtools.yang.data.api.schema.tree.ConflictingModificationAppliedException: Node was created by other transaction.
at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkWriteApplicable(SchemaAwareApplyOperation.java:174)
at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:128)
at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkChildPreconditions(AbstractNodeContainerModificationStrategy.java:305)
at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkMergeApplicable(AbstractNodeContainerModificationStrategy.java:313)
at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:131)
at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkChildPreconditions(AbstractNodeContainerModificationStrategy.java:305)
at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkMergeApplicable(AbstractNodeContainerModificationStrategy.java:313)
at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:131)
at org.opendaylight.yangtools.yang.data.impl.schema.tree.StructuralContainerModificationStrategy.checkApplicable(StructuralContainerModificationStrategy.java:99)
at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkChildPreconditions(AbstractNodeContainerModificationStrategy.java:305)
at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkTouchApplicable(AbstractNodeContainerModificationStrategy.java:288)
at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:125)
at org.opendaylight.yangtools.yang.data.impl.schema.tree.RootModificationApplyOperation.checkApplicable(RootModificationApplyOperation.java:72)
at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractDataTreeTip.validate(AbstractDataTreeTip.java:35)
at org.opendaylight.controller.cluster.datastore.ShardDataTree.lambda$processNextPendingTransaction$0(ShardDataTree.java:570)
... 28 more
2017-02-07 05:22:39,837 | INFO | rint Extender: 2 | FlowCapableTopologyProvider | 253 - org.opendaylight.openflowplugin.applications.topology-manager - 0.4.0.SNAPSHOT | FlowCapableTopologyProvider started
2017-02-07 05:22:39,843 | INFO | tAdminThread #19 | BlueprintBundleTracker | 192 - org.opendaylight.controller.blueprint - 0.6.0.SNAPSHOT | Blueprint container for bundle org.opendaylight.openflowplugin.applications.topology-manager_0.4.0.SNAPSHOT [253] was successfully created

[0] https://logs.opendaylight.org/releng/jenkins092/openflowplugin-csit-3node-periodic-bulkomatic-clustering-perf-daily-only-carbon/147/archives/log.html.gz#s1-s1-t6-k1-k1-k1
[1] https://logs.opendaylight.org/releng/jenkins092/openflowplugin-csit-3node-periodic-bulkomatic-clustering-perf-daily-only-carbon/147/archives/odl2_karaf.log.gz

OpenDaylight JIRA

[OPNFLWPLUG-849] "Initial topology export failed" while pushing 42-openflowplugin-Li.xml