[NETVIRT-994] OptimisticLockFailedException: selectNewNAPTSwitch...ConflictingModificationAppliedException: Node was created by other transaction Created: 14/Nov/17  Updated: 30/Nov/17  Resolved: 30/Nov/17

Status: Resolved
Project: netvirt
Component/s: General
Affects Version/s: Carbon
Fix Version/s: Carbon-SR3

Type: Bug Priority: Medium
Reporter: Sam Hague Assignee: Karthikeyan Krishnan
Resolution: Cannot Reproduce Votes: 0
Labels: csit:exception
Remaining Estimate: 0 minutes
Time Spent: 1 week
Original Estimate: Not Specified


 Description   

https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-pike-upstream-stateful-carbon/65/odl_1/odl1_karaf.log.gz

2017-11-13 07:40:20,655 | ERROR | eChangeHandler-0 | AsyncDataTreeChangeListenerBase  | 292 - org.opendaylight.genius.mdsalutil-api - 0.2.3.SNAPSHOT | Thread terminated due to uncaught exception: AsyncDataTreeChangeListenerBase-DataTreeChangeHandler-0
java.lang.RuntimeException: OptimisticLockFailedException{message=Optimistic lock failed., errorList=[RpcError [message=Optimistic lock failed., severity=ERROR, errorType=APPLICATION, tag=resource-denied, applicationTag=null, info=null, cause=org.opendaylight.yangtools.yang.data.api.schema.tree.ConflictingModificationAppliedException: Node was created by other transaction.]]}
	at org.opendaylight.genius.mdsalutil.MDSALUtil.syncWrite(MDSALUtil.java:586)[292:org.opendaylight.genius.mdsalutil-api:0.2.3.SNAPSHOT]
	at org.opendaylight.netvirt.natservice.internal.NAPTSwitchSelector.selectNewNAPTSwitch(NAPTSwitchSelector.java:78)[342:org.opendaylight.netvirt.natservice-impl:0.4.3.SNAPSHOT]
	at org.opendaylight.netvirt.natservice.internal.ExternalRoutersListener.getPrimaryNaptSwitch(ExternalRoutersListener.java:348)[342:org.opendaylight.netvirt.natservice-impl:0.4.3.SNAPSHOT]
	at org.opendaylight.netvirt.natservice.internal.ExternalRoutersListener.update(ExternalRoutersListener.java:1215)[342:org.opendaylight.netvirt.natservice-impl:0.4.3.SNAPSHOT]
	at org.opendaylight.netvirt.natservice.internal.ExternalRoutersListener.update(ExternalRoutersListener.java:140)[342:org.opendaylight.netvirt.natservice-impl:0.4.3.SNAPSHOT]
	at org.opendaylight.genius.datastoreutils.AsyncDataTreeChangeListenerBase$DataTreeChangeHandler.run(AsyncDataTreeChangeListenerBase.java:160)[292:org.opendaylight.genius.mdsalutil-api:0.2.3.SNAPSHOT]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)[:1.8.0_151]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)[:1.8.0_151]
	at java.lang.Thread.run(Thread.java:748)[:1.8.0_151]
Caused by: OptimisticLockFailedException{message=Optimistic lock failed., errorList=[RpcError [message=Optimistic lock failed., severity=ERROR, errorType=APPLICATION, tag=resource-denied, applicationTag=null, info=null, cause=org.opendaylight.yangtools.yang.data.api.schema.tree.ConflictingModificationAppliedException: Node was created by other transaction.]]}
	at org.opendaylight.controller.cluster.datastore.ShardDataTree.lambda$processNextPendingTransaction$0(ShardDataTree.java:751)[184:org.opendaylight.controller.sal-distributed-datastore:1.5.3.SNAPSHOT]
	at org.opendaylight.controller.cluster.datastore.ShardDataTree.processNextPending(ShardDataTree.java:789)[184:org.opendaylight.controller.sal-distributed-datastore:1.5.3.SNAPSHOT]
	at org.opendaylight.controller.cluster.datastore.ShardDataTree.processNextPendingTransaction(ShardDataTree.java:736)[184:org.opendaylight.controller.sal-distributed-datastore:1.5.3.SNAPSHOT]
	at org.opendaylight.controller.cluster.datastore.ShardDataTree.startCanCommit(ShardDataTree.java:819)[184:org.opendaylight.controller.sal-distributed-datastore:1.5.3.SNAPSHOT]
	at org.opendaylight.controller.cluster.datastore.SimpleShardDataTreeCohort.canCommit(SimpleShardDataTreeCohort.java:90)[184:org.opendaylight.controller.sal-distributed-datastore:1.5.3.SNAPSHOT]
	at org.opendaylight.controller.cluster.datastore.CohortEntry.canCommit(CohortEntry.java:97)[184:org.opendaylight.controller.sal-distributed-datastore:1.5.3.SNAPSHOT]
	at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator.handleCanCommit(ShardCommitCoordinator.java:236)[184:org.opendaylight.controller.sal-distributed-datastore:1.5.3.SNAPSHOT]
	at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator.handleReadyLocalTransaction(ShardCommitCoordinator.java:200)[184:org.opendaylight.controller.sal-distributed-datastore:1.5.3.SNAPSHOT]
	at org.opendaylight.controller.cluster.datastore.Shard.handleReadyLocalTransaction(Shard.java:675)[184:org.opendaylight.controller.sal-distributed-datastore:1.5.3.SNAPSHOT]
	at org.opendaylight.controller.cluster.datastore.Shard.handleNonRaftCommand(Shard.java:316)[184:org.opendaylight.controller.sal-distributed-datastore:1.5.3.SNAPSHOT]
	at org.opendaylight.controller.cluster.raft.RaftActor.handleCommand(RaftActor.java:270)[178:org.opendaylight.controller.sal-akka-raft:1.5.3.SNAPSHOT]
	at org.opendaylight.controller.cluster.common.actor.AbstractUntypedPersistentActor.onReceiveCommand(AbstractUntypedPersistentActor.java:44)[177:org.opendaylight.controller.sal-clustering-commons:1.5.3.SNAPSHOT]
	at akka.persistence.UntypedPersistentActor.onReceive(PersistentActor.scala:170)[165:com.typesafe.akka.persistence:2.4.18]
	at org.opendaylight.controller.cluster.common.actor.MeteringBehavior.apply(MeteringBehavior.java:104)[177:org.opendaylight.controller.sal-clustering-commons:1.5.3.SNAPSHOT]
	at akka.actor.ActorCell$$anonfun$become$1.applyOrElse(ActorCell.scala:544)[158:com.typesafe.akka.actor:2.4.18]
	at akka.actor.Actor$class.aroundReceive(Actor.scala:502)[158:com.typesafe.akka.actor:2.4.18]
	at akka.persistence.UntypedPersistentActor.akka$persistence$Eventsourced$$super$aroundReceive(PersistentActor.scala:168)[165:com.typesafe.akka.persistence:2.4.18]
	at akka.persistence.Eventsourced$$anon$1.stateReceive(Eventsourced.scala:727)[165:com.typesafe.akka.persistence:2.4.18]
	at akka.persistence.Eventsourced$class.aroundReceive(Eventsourced.scala:183)[165:com.typesafe.akka.persistence:2.4.18]
	at akka.persistence.UntypedPersistentActor.aroundReceive(PersistentActor.scala:168)[165:com.typesafe.akka.persistence:2.4.18]
	at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)[158:com.typesafe.akka.actor:2.4.18]
	at akka.actor.ActorCell.invoke(ActorCell.scala:495)[158:com.typesafe.akka.actor:2.4.18]
	at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)[158:com.typesafe.akka.actor:2.4.18]
	at akka.dispatch.Mailbox.run(Mailbox.scala:224)[158:com.typesafe.akka.actor:2.4.18]
	at akka.dispatch.Mailbox.exec(Mailbox.scala:234)[158:com.typesafe.akka.actor:2.4.18]
	at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)[154:org.scala-lang.scala-library:2.11.11.v20170413-090219-8a413ba7cc]
	at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)[154:org.scala-lang.scala-library:2.11.11.v20170413-090219-8a413ba7cc]
	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)[154:org.scala-lang.scala-library:2.11.11.v20170413-090219-8a413ba7cc]
	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)[154:org.scala-lang.scala-library:2.11.11.v20170413-090219-8a413ba7cc]
Caused by: org.opendaylight.yangtools.yang.data.api.schema.tree.ConflictingModificationAppliedException: Node was created by other transaction.
	at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkWriteApplicable(SchemaAwareApplyOperation.java:174)[81:org.opendaylight.yangtools.yang-data-impl:1.1.3.SNAPSHOT]
	at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:128)[81:org.opendaylight.yangtools.yang-data-impl:1.1.3.SNAPSHOT]
	at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkChildPreconditions(AbstractNodeContainerModificationStrategy.java:305)[81:org.opendaylight.yangtools.yang-data-impl:1.1.3.SNAPSHOT]
	at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkMergeApplicable(AbstractNodeContainerModificationStrategy.java:313)[81:org.opendaylight.yangtools.yang-data-impl:1.1.3.SNAPSHOT]
	at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:131)[81:org.opendaylight.yangtools.yang-data-impl:1.1.3.SNAPSHOT]
	at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkChildPreconditions(AbstractNodeContainerModificationStrategy.java:305)[81:org.opendaylight.yangtools.yang-data-impl:1.1.3.SNAPSHOT]
	at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkMergeApplicable(AbstractNodeContainerModificationStrategy.java:313)[81:org.opendaylight.yangtools.yang-data-impl:1.1.3.SNAPSHOT]
	at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:131)[81:org.opendaylight.yangtools.yang-data-impl:1.1.3.SNAPSHOT]
	at org.opendaylight.yangtools.yang.data.impl.schema.tree.StructuralContainerModificationStrategy.checkApplicable(StructuralContainerModificationStrategy.java:99)[81:org.opendaylight.yangtools.yang-data-impl:1.1.3.SNAPSHOT]
	at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkChildPreconditions(AbstractNodeContainerModificationStrategy.java:305)[81:org.opendaylight.yangtools.yang-data-impl:1.1.3.SNAPSHOT]
	at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkTouchApplicable(AbstractNodeContainerModificationStrategy.java:288)[81:org.opendaylight.yangtools.yang-data-impl:1.1.3.SNAPSHOT]
	at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:125)[81:org.opendaylight.yangtools.yang-data-impl:1.1.3.SNAPSHOT]
	at org.opendaylight.yangtools.yang.data.impl.schema.tree.RootModificationApplyOperation.checkApplicable(RootModificationApplyOperation.java:72)[81:org.opendaylight.yangtools.yang-data-impl:1.1.3.SNAPSHOT]
	at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractDataTreeTip.validate(AbstractDataTreeTip.java:35)[81:org.opendaylight.yangtools.yang-data-impl:1.1.3.SNAPSHOT]
	at org.opendaylight.controller.cluster.datastore.ShardDataTree.lambda$processNextPendingTransaction$0(ShardDataTree.java:743)[184:org.opendaylight.controller.sal-distributed-datastore:1.5.3.SNAPSHOT]
	... 28 more


 Comments   
Comment by Karthikeyan Krishnan [ 17/Nov/17 ]

Will start look into the issue

Comment by Karthikeyan Krishnan [ 24/Nov/17 ]

Reported issue is not been reproducible in recent Upstream CSIT jobs. Please refer the below link for the recent Upstream CSIT logs.
Still working on the fix to close this issue.

https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-pike-upstream-stateful-carbon/78/odl_1/odl1_karaf.log.gz
https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-pike-upstream-stateful-carbon/77/odl_1/odl1_karaf.log.gz
https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-pike-upstream-stateful-carbon/76/odl_1/odl1_karaf.log.gz

Comment by Karthikeyan Krishnan [ 30/Nov/17 ]

We monitored continuously reported exception from NAT module and from the past few weeks this issue was not re-producible. Hence closing this issue as of now. If this issue is occurring in future will re-open the bug again. 

Most recent logs:

=================

https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-pike-upstream-stateful-carbon/86/odl_1/odl1_karaf.log.gz

 

https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-pike-upstream-stateful-carbon/85/odl_1/odl1_karaf.log.gz

 

 

Comment by Karthikeyan Krishnan [ 30/Nov/17 ]
We monitored continuously reported exception from NAT module and from the past few weeks this issue was not re-producible. Hence closing this issue as of now. If this issue is occurring in future will re-open the bug again. 
 
Most recent logs:
=================
https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-pike-upstream-stateful-carbon/86/odl_1/odl1_karaf.log.gz
 
https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-pike-upstream-stateful-carbon/85/odl_1/odl1_karaf.log.gz
 
Generated at Wed Feb 07 20:22:58 UTC 2024 using Jira 8.20.10#820010-sha1:ace47f9899e9ee25d7157d59aa17ab06aee30d3d.