https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/NETVIRT-1260 netvirt <p><a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-openstack-queens-upstream-stateful-fluorine/421/odl_1/odl1_karaf.log.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-openstack-queens-upstream-stateful-fluorine/421/odl_1/odl1_karaf.log.gz</a></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>2018-05-11T17:06:10,067 | WARN | opendaylight-cluster-data-shard-dispatcher-32 | ShardDataTree | 246 - org.opendaylight.controller.sal-distributed-datastore - 1.8.0.SNAPSHOT | member-1-shard-default-config: Store Tx member-1-datastore-config-fe-0-txn-8784-0: Conflicting modification for path /(urn:ietf:params:xml:ns:yang:ietf-interfaces?revision=2014-05-08)interfaces/interface/interface[{(urn:ietf:params:xml:ns:yang:ietf-interfaces?revision=2014-05-08)name=ed2e5068-31bd-4bcd-9479-0860a9eba76e}]. 2018-05-11T17:06:10,083 | INFO | ForkJoinPool-1-worker-2 | InterfaceAddWorkerOnElanInterface | 367 - org.opendaylight.netvirt.elanmanager-impl - 0.7.0.SNAPSHOT | Handling elan interface bc5e0034-f81a-47ff-904f-49376d4312e7 add for elan 703b8c5f-3d4e-4ed2-b628-8860ba3de59e 2018-05-11T17:06:10,070 | ERROR | opendaylight-cluster-data-akka.actor.default-dispatcher-3 | LocalThreePhaseCommitCohort | 246 - org.opendaylight.controller.sal-distributed-datastore - 1.8.0.SNAPSHOT | Failed to prepare transaction member-1-datastore-config-fe-0-txn-8784-0 on backend org.opendaylight.mdsal.common.api.OptimisticLockFailedException: Optimistic lock failed. at org.opendaylight.controller.cluster.datastore.ShardDataTree.lambda$processNextPendingTransaction$0(ShardDataTree.java:740) ~[246:org.opendaylight.controller.sal-distributed-datastore:1.8.0.SNAPSHOT] at org.opendaylight.controller.cluster.datastore.ShardDataTree.processNextPending(ShardDataTree.java:778) [246:org.opendaylight.controller.sal-distributed-datastore:1.8.0.SNAPSHOT] at org.opendaylight.controller.cluster.datastore.ShardDataTree.processNextPendingTransaction(ShardDataTree.java:725) [246:org.opendaylight.controller.sal-distributed-datastore:1.8.0.SNAPSHOT] at org.opendaylight.controller.cluster.datastore.ShardDataTree.startCanCommit(ShardDataTree.java:808) [246:org.opendaylight.controller.sal-distributed-datastore:1.8.0.SNAPSHOT] at org.opendaylight.controller.cluster.datastore.SimpleShardDataTreeCohort.canCommit(SimpleShardDataTreeCohort.java:84) [246:org.opendaylight.controller.sal-distributed-datastore:1.8.0.SNAPSHOT] at org.opendaylight.controller.cluster.datastore.CohortEntry.canCommit(CohortEntry.java:97) [246:org.opendaylight.controller.sal-distributed-datastore:1.8.0.SNAPSHOT] at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator.handleCanCommit(ShardCommitCoordinator.java:236) [246:org.opendaylight.controller.sal-distributed-datastore:1.8.0.SNAPSHOT] at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator.handleReadyLocalTransaction(ShardCommitCoordinator.java:200) [246:org.opendaylight.controller.sal-distributed-datastore:1.8.0.SNAPSHOT] at org.opendaylight.controller.cluster.datastore.Shard.handleReadyLocalTransaction(Shard.java:731) [246:org.opendaylight.controller.sal-distributed-datastore:1.8.0.SNAPSHOT] at org.opendaylight.controller.cluster.datastore.Shard.handleNonRaftCommand(Shard.java:333) [246:org.opendaylight.controller.sal-distributed-datastore:1.8.0.SNAPSHOT] at org.opendaylight.controller.cluster.raft.RaftActor.handleCommand(RaftActor.java:270) [230:org.opendaylight.controller.sal-akka-raft:1.8.0.SNAPSHOT] at org.opendaylight.controller.cluster.common.actor.AbstractUntypedPersistentActor.onReceiveCommand(AbstractUntypedPersistentActor.java:44) [238:org.opendaylight.controller.sal-clustering-commons:1.8.0.SNAPSHOT] at akka.persistence.UntypedPersistentActor.onReceive(PersistentActor.scala:275) [43:com.typesafe.akka.persistence:2.5.11] at org.opendaylight.controller.cluster.common.actor.MeteringBehavior.apply(MeteringBehavior.java:104) [238:org.opendaylight.controller.sal-clustering-commons:1.8.0.SNAPSHOT] at akka.actor.ActorCell$$anonfun$become$1.applyOrElse(ActorCell.scala:608) [40:com.typesafe.akka.actor:2.5.11] at akka.actor.Actor.aroundReceive(Actor.scala:517) [40:com.typesafe.akka.actor:2.5.11] at akka.actor.Actor.aroundReceive$(Actor.scala:515) [40:com.typesafe.akka.actor:2.5.11] at akka.persistence.UntypedPersistentActor.akka$persistence$Eventsourced$$super$aroundReceive(PersistentActor.scala:273) [43:com.typesafe.akka.persistence:2.5.11] at akka.persistence.Eventsourced$$anon$1.stateReceive(Eventsourced.scala:691) [43:com.typesafe.akka.persistence:2.5.11] at akka.persistence.Eventsourced.aroundReceive(Eventsourced.scala:192) [43:com.typesafe.akka.persistence:2.5.11] at akka.persistence.Eventsourced.aroundReceive$(Eventsourced.scala:191) [43:com.typesafe.akka.persistence:2.5.11] at akka.persistence.UntypedPersistentActor.aroundReceive(PersistentActor.scala:273) [43:com.typesafe.akka.persistence:2.5.11] at akka.actor.ActorCell.receiveMessage(ActorCell.scala:590) [40:com.typesafe.akka.actor:2.5.11] at akka.actor.ActorCell.invoke(ActorCell.scala:559) [40:com.typesafe.akka.actor:2.5.11] at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257) [40:com.typesafe.akka.actor:2.5.11] at akka.dispatch.Mailbox.run(Mailbox.scala:224) [40:com.typesafe.akka.actor:2.5.11] at akka.dispatch.Mailbox.exec(Mailbox.scala:234) [40:com.typesafe.akka.actor:2.5.11] at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) [40:com.typesafe.akka.actor:2.5.11] at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) [40:com.typesafe.akka.actor:2.5.11] at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) [40:com.typesafe.akka.actor:2.5.11] at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) [40:com.typesafe.akka.actor:2.5.11] Caused by: org.opendaylight.yangtools.yang.data.api.schema.tree.ConflictingModificationAppliedException: Node children was modified by other transaction at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkConflicting(SchemaAwareApplyOperation.java:81) ~[446:org.opendaylight.yangtools.yang-data-impl:2.0.3] at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkNotConflicting(SchemaAwareApplyOperation.java:114) ~[446:org.opendaylight.yangtools.yang-data-impl:2.0.3] at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkWriteApplicable(SchemaAwareApplyOperation.java:178) ~[446:org.opendaylight.yangtools.yang-data-impl:2.0.3] at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:135) ~[446:org.opendaylight.yangtools.yang-data-impl:2.0.3] at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkChildPreconditions(AbstractNodeContainerModificationStrategy.java:307) ~[446:org.opendaylight.yangtools.yang-data-impl:2.0.3] at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkMergeApplicable(AbstractNodeContainerModificationStrategy.java:315) ~[446:org.opendaylight.yangtools.yang-data-impl:2.0.3] at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:138) ~[446:org.opendaylight.yangtools.yang-data-impl:2.0.3] at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkChildPreconditions(AbstractNodeContainerModificationStrategy.java:307) ~[446:org.opendaylight.yangtools.yang-data-impl:2.0.3] at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkTouchApplicable(AbstractNodeContainerModificationStrategy.java:290) ~[446:org.opendaylight.yangtools.yang-data-impl:2.0.3] at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:132) ~[446:org.opendaylight.yangtools.yang-data-impl:2.0.3] at org.opendaylight.yangtools.yang.data.impl.schema.tree.StructuralContainerModificationStrategy.checkApplicable(StructuralContainerModificationStrategy.java:101) ~[446:org.opendaylight.yangtools.yang-data-impl:2.0.3] at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkChildPreconditions(AbstractNodeContainerModificationStrategy.java:307) ~[446:org.opendaylight.yangtools.yang-data-impl:2.0.3] at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkTouchApplicable(AbstractNodeContainerModificationStrategy.java:290) ~[446:org.opendaylight.yangtools.yang-data-impl:2.0.3] at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:132) ~[446:org.opendaylight.yangtools.yang-data-impl:2.0.3] at org.opendaylight.yangtools.yang.data.impl.schema.tree.RootModificationApplyOperation.checkApplicable(RootModificationApplyOperation.java:72) ~[446:org.opendaylight.yangtools.yang-data-impl:2.0.3] at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractDataTreeTip.validate(AbstractDataTreeTip.java:35) ~[446:org.opendaylight.yangtools.yang-data-impl:2.0.3] at org.opendaylight.controller.cluster.datastore.ShardDataTree.lambda$processNextPendingTransaction$0(ShardDataTree.java:732) ~[246:org.opendaylight.controller.sal-distributed-datastore:1.8.0.SNAPSHOT] ... 30 more </pre> </div></div> NETVIRT-1260

OLFE: Conflicting modification for path... interfaces/interface/interface... name=ed2e5068-31bd-4bcd-9479-0860a9eba76e

Bug Medium Resolved Cannot Reproduce Abhinav Gupta Sam Hague csit:exception Tue, 15 May 2018 21:29:55 +0000 Tue, 19 Nov 2019 18:34:39 +0000 Wed, 3 Oct 2018 19:27:16 +0000 0 6 <p>One thing I always wanted to ask is, when this OptimistickLockFailedException comes, it would be good if we can atleast get the path at which the exception has come. It is difficult to track from logs otherwise to understand which datastore write exactly resulted in this error.</p> <p>We are just assuming things from the immediate log before the exception, which may not be always true.</p> <p> </p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=vorburger" class="user-hover" rel="vorburger">vorburger</a> : Any help can be done in this area?</p> <p>Assuming that the conflicting modification log immediately above this is the reason for this exception :</p> <p>2018-05-11T17:06:10,067 | WARN | opendaylight-cluster-data-shard-dispatcher-32 | ShardDataTree | 246 - org.opendaylight.controller.sal-distributed-datastore - 1.8.0.SNAPSHOT | member-1-shard-default-config: Store Tx member-1-datastore-config-fe-0-txn-8784-0: Conflicting modification for path /(urn:ietf:params:xml:ns:yang:ietf-interfaces?revision=2014-05-08)interfaces/interface/interface<span class="error">&#91;\{(urn:ietf:params:xml:ns:yang:ietf-interfaces?revision=2014-05-08)name=ed2e5068-31bd-4bcd-9479-0860a9eba76e}&#93;</span>.</p> <p>do we have any recent change in NeutronPortChangeListener in neutronvpn that has recently landed?</p> <p>&gt; Any help can be done in this area?</p> <p><a href="https://jira.opendaylight.org/browse/CONTROLLER-1830" title="Add path to exception logs" class="issue-link" data-issue-key="CONTROLLER-1830"><del>CONTROLLER-1830</del></a> is taking care of this, from what I understood.</p> <p>still seen: <a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-openstack-queens-upstream-stateful-fluorine/509/odl_1/odl1_karaf.log.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-openstack-queens-upstream-stateful-fluorine/509/odl_1/odl1_karaf.log.gz</a></p> <p>I had a look at the neutronvpn code.</p> <p>The reason why it  throws exception even for single node is that DJC in neutron vpn uses a different key than the DJC in interface-manager, and hence the error is thrown.</p> <p>Now, even if we fix the DJC key, the solution will not work for cluster.</p> <p>I can think of two solutions :</p> <p> </p> <ul> <li>Use DataTreeEventCallbackRegistrar to make one of the two guys wait till the other guy pushes all information.(This utility currently does not work in Cluster, see : <a href="https://jira.opendaylight.org/browse/GENIUS-141" title="DataTreeEventCallbackRegistrar does not work in Cluster in all scenarios" class="issue-link" data-issue-key="GENIUS-141"><del>GENIUS-141</del></a>, which I am trying to fix first. Also not quite sure whether this will bring in any performance bottlenecks).</li> <li>Move updating parent-ref logic back to neutron-vpn. Neutron-vpn can listen for ovsdb event to get the external-id information and do update parent-interface, which will easily align with single writer principle.</li> </ul> <p>We analyzed logs and below are our observations from ACL point of view</p> <p>(1) From the logs OptimisticLockFailedException is observed for ietf-interfaces path as shown below (with latest logs attached):</p> <p> org.opendaylight.mdsal.common.api.OptimisticLockFailedException: Optimistic lock failed for path /(urn:ietf:params:xml:ns:yang:ietf-interfaces?revision=2014-05-08)interfaces/interface/interface<span class="error">&#91;\{(urn:ietf:params:xml:ns:yang:ietf-interfaces?revision=2014-05-08)name=ecb03cb6-8a45-4b07-b760-3066f6017a5d}&#93;</span></p> <p> (2) ACL only reads ietf-interfaces for doing validations; but does not modify interfaces. As it doesn’t modify interfaces, the  OptimisticLockFailedException observed above is not from ACL module.</p> <p> </p> <p>I also discussed about this with Abhinav from neutrovpn point of view. Below are the details:</p> <ol> <li>In oxygen and fluorine, interface manager is updating the parent refs.</li> <li>As Neutronvpn creates/deletes the interface with jobCoordinator, ConflictingModificationAppliedException would not be in Neutronvpn.</li> <li>Also SG addition/deletion to a port is modification of one of the parameter of neutron port itself. As this is already handled with jobCoordinator, ConflictingModificationAppliedException would not be because of SG updates as well.</li> </ol> <p>Modification of SGs (like addition or deletion of rules): Handled independent of ietf-interfaces both in Neutronvpn and ACL modules.</p> <p>Assigning bug to Faseela for analyzing further from interface manger point of view.</p> <p>Hi Faseela,</p> <p>Regarding the two possible solutions you’ve suggested:</p> <ul> <li>Use DataTreeEventCallbackRegistrar to make one of the two guys wait till the other guy pushes all information.(This utility currently does not work in Cluster, see : <a href="https://jira.opendaylight.org/browse/GENIUS-141" title="DataTreeEventCallbackRegistrar does not work in Cluster in all scenarios" class="issue-link" data-issue-key="GENIUS-141"><del>GENIUS-141</del></a>, which I am trying to fix first. Also not quite sure whether this will bring in any performance bottlenecks).</li> <li><span class="error">&#91;Abhinav&#93;</span> Since this is at per port level, this could rightly bring in performance bottlenecks and we may want a better solution.</li> </ul> <p> </p> <ul> <li>Move updating parent-ref logic back to neutron-vpn. Neutron-vpn can listen for ovsdb event to get the external-id information and do update parent-interface, which will easily align with single writer principle.</li> <li><span class="error">&#91;Abhinav&#93;</span>We’ve always tried to keep nvpn decoupled from southbound logic. This could be a last resort solution if nothing else fits.</li> </ul> <p> </p> <p>So currently,</p> <p>NeutronVPN: Creates/deletes the interface, updates (adds) InterfaceACL augmentation for security updates</p> <p>Inf-Mgr          : Updates the parentRefs</p> <p> </p> <p>Since inf-mgr has the entire data to create inf upon receiving exernal-id info from OVSDB, can we create the inf the first time then itself, as well as handle deletes based on the same? ACL aug updates today are doing a read on the entire inf object and then doing a PUT (looks like a leftover PUT from when the entire object was fetched for updating parent-refs). This can be changed to MERGE as well as we can write directly on the augmentation path identifier.</p> <p> </p> <p>This would align with the single writer principle. What do you think?</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=AbhinavGupta" class="user-hover" rel="AbhinavGupta">AbhinavGupta</a>,</p> <p>  Interface-manager will have trouble in understanding the interface-type, trunk-port or sub-port as I don’t see that information coming from southbound. If this can be supplied by nova while creating the VM, it can be done. Could you please confirm?</p> <p>Else the solution you have proposed is not feasible. So please pick one solution from the 2 I have given <img class="emoticon" src="https://jira.opendaylight.org/images/icons/emoticons/wink.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>Thanks,</p> <p>Faseela</p> <p>Let us decide on what will be the preferred approach from the 2 solutions specified. Assigning the bug to Abhinav to decide on the solution. Based on whether the change is needed in neutron-vpn or interface-manager, we can change the assignee later on.</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=shague" class="user-hover" rel="shague">shague</a> : One quick question, is this yet another exception coming in the log, or are there functional impacts.</p> <p>When I take a look at the code, we are using DJC retries, and if two threads are doing transaction.merge(), for the initial time we might get the exception, but subsequent retries would have gone through. Is that the case?</p> <p>If so, do we just want to enhance DJC not print the whole exception stack trace for initial trials, and only do it if all retries fail?</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=vorburger" class="user-hover" rel="vorburger">vorburger</a> : Does this seem valid?</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=k.faseela" class="user-hover" rel="k.faseela">k.faseela</a> I am pretty sure that I have already done exactly what you propose quite a while ago, check out:</p> <ul> <li><a href="https://github.com/opendaylight/infrautils/commit/77cc4ccfc65b98d7dc4b0b24052a60f9e77c065b#diff-8dc9ad74499a7d8d42ed6a98b2787939" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/opendaylight/infrautils/commit/77cc4ccfc65b98d7dc4b0b24052a60f9e77c065b#diff-8dc9ad74499a7d8d42ed6a98b2787939</a></li> <li><a href="https://github.com/opendaylight/infrautils/commit/f7dc06b9515c915f2a0dd6ec1deab759f535454d#diff-8dc9ad74499a7d8d42ed6a98b2787939" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/opendaylight/infrautils/commit/f7dc06b9515c915f2a0dd6ec1deab759f535454d#diff-8dc9ad74499a7d8d42ed6a98b2787939</a></li> <li><a href="https://github.com/opendaylight/infrautils/commit/a05719f6ea7b9afacf502d5587c62184b55218a8#diff-8dc9ad74499a7d8d42ed6a98b2787939" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/opendaylight/infrautils/commit/a05719f6ea7b9afacf502d5587c62184b55218a8#diff-8dc9ad74499a7d8d42ed6a98b2787939</a></li> </ul> <p>If you have reason to believe that this is not working as we would like it to, can you open a new Jira in INFRAUTILS with details of a case where you think it is printing the whole exception stack trace for initial trials? I don't see how.</p> <p>Looked into Karaf logs from build 510 to 539 for a few builds randomly. This exception doesn't show up.<br/> 509 was the last build where this exception was seen, and from the Karaf logs, looks like to be a conflict between SG update from NeutronVPN and parent ref update from inf-mgr.</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=k.faseela" class="user-hover" rel="k.faseela">k.faseela</a>, the exception here, as per build 509 occurs only once throughout the log where LocalThreePhaseCommitCohort prints an ERROR message.<br/> Looks like the 2nd retry succeeded, and there's no functionality impact here.</p> <p> </p> <p>Hi <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=shague" class="user-hover" rel="shague">shague</a>, as this exception was last seen in build 509, do we need to keep tracking this one?</p> <p> </p> <p> </p> Relates NETVIRT-1259 Development Rank 0|i03ekv: