[NETVIRT-993] VrfEntryListener: OptimisticLockFailedException: VrfEntryListener$$Lambda$639/784243272@562edc97...ConflictingModificationAppliedException: Node was created by other transaction Created: 14/Nov/17  Updated: 06/Apr/18  Resolved: 06/Apr/18

Status: Resolved
Project: netvirt
Component/s: General
Affects Version/s: Nitrogen, Carbon, Oxygen
Fix Version/s: None

Type: Bug Priority: Medium
Reporter: Sam Hague Assignee: Hanamantagoud Kandagal
Resolution: Cannot Reproduce Votes: 0
Labels: csit:exception
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Relates
relates to NETVIRT-972 Conflicting modification for path...n... Resolved
relates to NETVIRT-987 Conflicting modification for path... ... Resolved

 Description   

https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-pike-upstream-stateful-carbon/68/odl_1/odl1_karaf.log.gz

2017-11-14 02:41:16,574 | WARN  | CommitFutures-7  | DataStoreJobCoordinator          | 292 - org.opendaylight.genius.mdsalutil-api - 0.2.3.SNAPSHOT | Job: JobEntry{key='FIB-8d413cf8-8a74-45ed-a919-74f6803d4c26-30.0.0.2/32', mainWorker=org.opendaylight.netvirt.fibmanager.VrfEntryListener$$Lambda$639/784243272@562edc97, rollbackWorker=null, retryCount=3, futures=[org.opendaylight.controller.cluster.databroker.ConcurrentDOMDataBroker$AsyncNotifyingSettableFuture@5211428a]} failed
OptimisticLockFailedException{message=Optimistic lock failed., errorList=[RpcError [message=Optimistic lock failed., severity=ERROR, errorType=APPLICATION, tag=resource-denied, applicationTag=null, info=null, cause=org.opendaylight.yangtools.yang.data.api.schema.tree.ConflictingModificationAppliedException: Node was created by other transaction.]]}
	at org.opendaylight.controller.cluster.datastore.ShardDataTree.lambda$processNextPendingTransaction$0(ShardDataTree.java:751)[184:org.opendaylight.controller.sal-distributed-datastore:1.5.3.SNAPSHOT]
	at org.opendaylight.controller.cluster.datastore.ShardDataTree.processNextPending(ShardDataTree.java:789)[184:org.opendaylight.controller.sal-distributed-datastore:1.5.3.SNAPSHOT]
	at org.opendaylight.controller.cluster.datastore.ShardDataTree.processNextPendingTransaction(ShardDataTree.java:736)[184:org.opendaylight.controller.sal-distributed-datastore:1.5.3.SNAPSHOT]
	at org.opendaylight.controller.cluster.datastore.ShardDataTree.startCanCommit(ShardDataTree.java:819)[184:org.opendaylight.controller.sal-distributed-datastore:1.5.3.SNAPSHOT]
	at org.opendaylight.controller.cluster.datastore.SimpleShardDataTreeCohort.canCommit(SimpleShardDataTreeCohort.java:90)[184:org.opendaylight.controller.sal-distributed-datastore:1.5.3.SNAPSHOT]
	at org.opendaylight.controller.cluster.datastore.CohortEntry.canCommit(CohortEntry.java:97)[184:org.opendaylight.controller.sal-distributed-datastore:1.5.3.SNAPSHOT]
	at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator.handleCanCommit(ShardCommitCoordinator.java:236)[184:org.opendaylight.controller.sal-distributed-datastore:1.5.3.SNAPSHOT]
	at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator.handleReadyLocalTransaction(ShardCommitCoordinator.java:200)[184:org.opendaylight.controller.sal-distributed-datastore:1.5.3.SNAPSHOT]
	at org.opendaylight.controller.cluster.datastore.Shard.handleReadyLocalTransaction(Shard.java:675)[184:org.opendaylight.controller.sal-distributed-datastore:1.5.3.SNAPSHOT]
	at org.opendaylight.controller.cluster.datastore.Shard.handleNonRaftCommand(Shard.java:316)[184:org.opendaylight.controller.sal-distributed-datastore:1.5.3.SNAPSHOT]
	at org.opendaylight.controller.cluster.raft.RaftActor.handleCommand(RaftActor.java:270)[178:org.opendaylight.controller.sal-akka-raft:1.5.3.SNAPSHOT]
	at org.opendaylight.controller.cluster.common.actor.AbstractUntypedPersistentActor.onReceiveCommand(AbstractUntypedPersistentActor.java:44)[177:org.opendaylight.controller.sal-clustering-commons:1.5.3.SNAPSHOT]
	at akka.persistence.UntypedPersistentActor.onReceive(PersistentActor.scala:170)[165:com.typesafe.akka.persistence:2.4.18]
	at org.opendaylight.controller.cluster.common.actor.MeteringBehavior.apply(MeteringBehavior.java:104)[177:org.opendaylight.controller.sal-clustering-commons:1.5.3.SNAPSHOT]
	at akka.actor.ActorCell$$anonfun$become$1.applyOrElse(ActorCell.scala:544)[158:com.typesafe.akka.actor:2.4.18]
	at akka.actor.Actor$class.aroundReceive(Actor.scala:502)[158:com.typesafe.akka.actor:2.4.18]
	at akka.persistence.UntypedPersistentActor.akka$persistence$Eventsourced$$super$aroundReceive(PersistentActor.scala:168)[165:com.typesafe.akka.persistence:2.4.18]
	at akka.persistence.Eventsourced$$anon$1.stateReceive(Eventsourced.scala:727)[165:com.typesafe.akka.persistence:2.4.18]
	at akka.persistence.Eventsourced$class.aroundReceive(Eventsourced.scala:183)[165:com.typesafe.akka.persistence:2.4.18]
	at akka.persistence.UntypedPersistentActor.aroundReceive(PersistentActor.scala:168)[165:com.typesafe.akka.persistence:2.4.18]
	at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)[158:com.typesafe.akka.actor:2.4.18]
	at akka.actor.ActorCell.invoke(ActorCell.scala:495)[158:com.typesafe.akka.actor:2.4.18]
	at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)[158:com.typesafe.akka.actor:2.4.18]
	at akka.dispatch.Mailbox.run(Mailbox.scala:224)[158:com.typesafe.akka.actor:2.4.18]
	at akka.dispatch.Mailbox.exec(Mailbox.scala:234)[158:com.typesafe.akka.actor:2.4.18]
	at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)[154:org.scala-lang.scala-library:2.11.11.v20170413-090219-8a413ba7cc]
	at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)[154:org.scala-lang.scala-library:2.11.11.v20170413-090219-8a413ba7cc]
	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)[154:org.scala-lang.scala-library:2.11.11.v20170413-090219-8a413ba7cc]
	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)[154:org.scala-lang.scala-library:2.11.11.v20170413-090219-8a413ba7cc]
Caused by: org.opendaylight.yangtools.yang.data.api.schema.tree.ConflictingModificationAppliedException: Node was created by other transaction.
	at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkWriteApplicable(SchemaAwareApplyOperation.java:174)[81:org.opendaylight.yangtools.yang-data-impl:1.1.3.SNAPSHOT]
	at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:128)[81:org.opendaylight.yangtools.yang-data-impl:1.1.3.SNAPSHOT]
	at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkChildPreconditions(AbstractNodeContainerModificationStrategy.java:305)[81:org.opendaylight.yangtools.yang-data-impl:1.1.3.SNAPSHOT]
	at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkMergeApplicable(AbstractNodeContainerModificationStrategy.java:313)[81:org.opendaylight.yangtools.yang-data-impl:1.1.3.SNAPSHOT]
	at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:131)[81:org.opendaylight.yangtools.yang-data-impl:1.1.3.SNAPSHOT]
	at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkChildPreconditions(AbstractNodeContainerModificationStrategy.java:305)[81:org.opendaylight.yangtools.yang-data-impl:1.1.3.SNAPSHOT]
	at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkMergeApplicable(AbstractNodeContainerModificationStrategy.java:313)[81:org.opendaylight.yangtools.yang-data-impl:1.1.3.SNAPSHOT]
	at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:131)[81:org.opendaylight.yangtools.yang-data-impl:1.1.3.SNAPSHOT]
	at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkChildPreconditions(AbstractNodeContainerModificationStrategy.java:305)[81:org.opendaylight.yangtools.yang-data-impl:1.1.3.SNAPSHOT]
	at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkMergeApplicable(AbstractNodeContainerModificationStrategy.java:313)[81:org.opendaylight.yangtools.yang-data-impl:1.1.3.SNAPSHOT]
	at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:131)[81:org.opendaylight.yangtools.yang-data-impl:1.1.3.SNAPSHOT]
	at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkChildPreconditions(AbstractNodeContainerModificationStrategy.java:305)[81:org.opendaylight.yangtools.yang-data-impl:1.1.3.SNAPSHOT]
	at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkMergeApplicable(AbstractNodeContainerModificationStrategy.java:313)[81:org.opendaylight.yangtools.yang-data-impl:1.1.3.SNAPSHOT]
	at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:131)[81:org.opendaylight.yangtools.yang-data-impl:1.1.3.SNAPSHOT]
	at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkChildPreconditions(AbstractNodeContainerModificationStrategy.java:305)[81:org.opendaylight.yangtools.yang-data-impl:1.1.3.SNAPSHOT]
	at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkMergeApplicable(AbstractNodeContainerModificationStrategy.java:313)[81:org.opendaylight.yangtools.yang-data-impl:1.1.3.SNAPSHOT]
	at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:131)[81:org.opendaylight.yangtools.yang-data-impl:1.1.3.SNAPSHOT]
	at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkChildPreconditions(AbstractNodeContainerModificationStrategy.java:305)[81:org.opendaylight.yangtools.yang-data-impl:1.1.3.SNAPSHOT]
	at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkMergeApplicable(AbstractNodeContainerModificationStrategy.java:313)[81:org.opendaylight.yangtools.yang-data-impl:1.1.3.SNAPSHOT]
	at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:131)[81:org.opendaylight.yangtools.yang-data-impl:1.1.3.SNAPSHOT]
	at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkChildPreconditions(AbstractNodeContainerModificationStrategy.java:305)[81:org.opendaylight.yangtools.yang-data-impl:1.1.3.SNAPSHOT]
	at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkMergeApplicable(AbstractNodeContainerModificationStrategy.java:313)[81:org.opendaylight.yangtools.yang-data-impl:1.1.3.SNAPSHOT]
	at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:131)[81:org.opendaylight.yangtools.yang-data-impl:1.1.3.SNAPSHOT]
	at org.opendaylight.yangtools.yang.data.impl.schema.tree.StructuralContainerModificationStrategy.checkApplicable(StructuralContainerModificationStrategy.java:99)[81:org.opendaylight.yangtools.yang-data-impl:1.1.3.SNAPSHOT]
	at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkChildPreconditions(AbstractNodeContainerModificationStrategy.java:305)[81:org.opendaylight.yangtools.yang-data-impl:1.1.3.SNAPSHOT]
	at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkTouchApplicable(AbstractNodeContainerModificationStrategy.java:288)[81:org.opendaylight.yangtools.yang-data-impl:1.1.3.SNAPSHOT]
	at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:125)[81:org.opendaylight.yangtools.yang-data-impl:1.1.3.SNAPSHOT]
	at org.opendaylight.yangtools.yang.data.impl.schema.tree.RootModificationApplyOperation.checkApplicable(RootModificationApplyOperation.java:72)[81:org.opendaylight.yangtools.yang-data-impl:1.1.3.SNAPSHOT]
	at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractDataTreeTip.validate(AbstractDataTreeTip.java:35)[81:org.opendaylight.yangtools.yang-data-impl:1.1.3.SNAPSHOT]
	at org.opendaylight.controller.cluster.datastore.ShardDataTree.lambda$processNextPendingTransaction$0(ShardDataTree.java:743)[184:org.opendaylight.controller.sal-distributed-datastore:1.5.3.SNAPSHOT]
	... 28 more

2017-11-14 02:41:16,574 | WARN  | CommitFutures-7  | DataStoreJobCoordinator          | 292 - org.opendaylight.genius.mdsalutil-api - 0.2.3.SNAPSHOT | Job: JobEntry{key='FIB-8d413cf8-8a74-45ed-a919-74f6803d4c26-30.0.0.2/32', mainWorker=org.opendaylight.netvirt.fibmanager.VrfEntryListener$$Lambda$639/784243272@562edc97, rollbackWorker=null, retryCount=3, futures=[org.opendaylight.controller.cluster.databroker.ConcurrentDOMDataBroker$AsyncNotifyingSettableFuture@5211428a]} failed
OptimisticLockFailedException{message=Optimistic lock failed., errorList=[RpcError [message=Optimistic lock failed., severity=ERROR, errorType=APPLICATION, tag=resource-denied, applicationTag=null, info=null, cause=org.opendaylight.yangtools.yang.data.api.schema.tree.ConflictingModificationAppliedException: Node was created by other transaction.]]}
	at org.opendaylight.controller.cluster.datastore.ShardDataTree.lambda$processNextPendingTransaction$0(ShardDataTree.java:751)[184:org.opendaylight.controller.sal-distributed-datastore:1.5.3.SNAPSHOT]


 Comments   
Comment by Sam Hague [ 16/Nov/17 ]

Still seen: https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-ocata-gate-stateful-oxygen/322/odl_1/odl1_karaf.log.gz

Comment by Sam Hague [ 28/Nov/17 ]

Still seen: https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-ocata-upstream-stateful-nitrogen/489/odl_1/odl1_karaf.log.gz

2017-11-28 10:30:52,804 | WARN  | CommitFutures-9  | DataStoreJobCoordinator          | 245 - org.opendaylight.genius.mdsalutil-api - 0.3.1 | Job: JobEntry{key='FIB-9391cf15-ed2d-413d-bafb-cbc98a39c55d-10.1.1.12/32', mainWorker=org.opendaylight.netvirt.fibmanager.VrfEntryListener$$Lambda$755/371884248@69280da7, rollbackWorker=null, retryCount=3, futures=[org.opendaylight.controller.cluster.databroker.ConcurrentDOMDataBroker$AsyncNotifyingSettableFuture@6d17945a]} failed
OptimisticLockFailedException{message=Optimistic lock failed., errorList=[RpcError [message=Optimistic lock failed., severity=ERROR, errorType=APPLICATION, tag=resource-denied, applicationTag=null, info=null, cause=org.opendaylight.yangtools.yang.data.api.schema.tree.ConflictingModificationAppliedException: Node was created by other transaction.]]}
Comment by Sam Hague [ 05/Dec/17 ]

Still seen on all branches: https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-ocata-upstream-stateful-oxygen/483/odl_1/odl1_exception.log.gz

https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-ocata-upstream-stateful-carbon/217/odl_1/odl1_exception.log.gz

Comment by Sam Hague [ 12/Dec/17 ]

Still seen:

oxygen: https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-ocata-upstream-stateful-oxygen/523/odl_1/odl1_karaf.log.gz

2017-12-12 03:35:54,146 | INFO  | nsole user karaf | core                             | 112 - org.apache.karaf.log.core - 4.0.10 | ROBOT MESSAGE: Starting test Associate L3VPN To Routers
...
2017-12-12 03:35:57,327 | WARN  | CommitFutures-5  | JobCoordinatorImpl               | 261 - org.opendaylight.infrautils.jobcoordinator-impl - 1.3.0.SNAPSHOT | Job: JobEntry{key='FIB-2200:2-20.1.1.2/32', mainWorker=org.opendaylight.netvirt.fibmanager.VrfEntryListener$$Lambda$840/681598788@26bf2cb4, rollbackWorker=null, retryCount=3, futures=[org.opendaylight.controller.cluster.databroker.ConcurrentDOMDataBroker$AsyncNotifyingSettableFuture@4b536376]} failed
OptimisticLockFailedException{message=Optimistic lock failed., errorList=[RpcError [message=Optimistic lock failed., severity=ERROR, errorType=APPLICATION, tag=resource-denied, applicationTag=null, info=null, cause=org.opendaylight.yangtools.yang.data.api.schema.tree.ConflictingModificationAppliedException: Node was created by other transaction.]]}

nitrogen: https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-ocata-upstream-stateful-nitrogen/574/odl_1/odl1_karaf.log.gz

2017-12-12 05:23:37,330 | INFO  | nsole user karaf | core                             | 112 - org.apache.karaf.log.core - 4.0.10 | ROBOT MESSAGE: Starting test Add Interfaces To Router
...
2017-12-12 05:23:40,606 | WARN  | CommitFutures-8  | DataStoreJobCoordinator          | 245 - org.opendaylight.genius.mdsalutil-api - 0.3.2.SNAPSHOT | Job: JobEntry{key='FIB-100008-136950895882886', mainWorker=org.opendaylight.netvirt.fibmanager.VrfEntryListener$$Lambda$795/194505305@688062b5, rollbackWorker=null, retryCount=0, futures=[org.opendaylight.controller.cluster.databroker.ConcurrentDOMDataBroker$AsyncNotifyingSettableFuture@6cbfeb7d]} failed
OptimisticLockFailedException{message=Optimistic lock failed., errorList=[RpcError [message=Optimistic lock failed., severity=ERROR, errorType=APPLICATION, tag=resource-denied, applicationTag=null, info=null, cause=org.opendaylight.yangtools.yang.data.api.schema.tree.ConflictingModificationAppliedException: Node was created by other transaction.]]}
Generated at Wed Feb 07 20:22:58 UTC 2024 using Jira 8.20.10#820010-sha1:ace47f9899e9ee25d7157d59aa17ab06aee30d3d.