https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/NEUTRON-157 neutron <p><a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-openstack-queens-upstream-stateful-oxygen/220/odl_1/odl1_karaf.log.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-openstack-queens-upstream-stateful-oxygen/220/odl_1/odl1_karaf.log.gz</a></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>2018-03-10T11:23:04,503 | WARN | opendaylight-cluster-data-shard-dispatcher-245 | ShardDataTree | 230 - org.opendaylight.controller.sal-distributed-datastore - 1.7.0.SNAPSHOT | member-1-shard-default-config: Store Tx member-1-datastore-config-fe-0-txn-35013-0: Conflicting modification for path /(urn:opendaylight:neutron?revision=2015-07-12)neutron/networks/network/network[{(urn:opendaylight:neutron?revision=2015-07-12)uuid=b674297c-3ae3-4940-b3b8-149cb2da8161}]. 2018-03-10T11:23:04,504 | WARN | qtp1247963896-1087 | AbstractTranscriberInterface | 372 - org.opendaylight.neutron.transcriber - 0.10.0.SNAPSHOT | Got OptimisticLockFailedException - b674297c-3ae3-4940-b3b8-149cb2da8161 NeutronNetwork [networkUUID=b674297c-3ae3-4940-b3b8-149cb2da8161, networkName=tempest-BulkNetworkOpsTest-test-network-157138568, adminStateUp=true, shared=false, tenantID=7268bcf662294cdfa8fdfe71b9d47934, routerExternal=false, providerNetworkType=null, providerPhysicalNetwork=null, providerSegmentationID=null, status=null, qosPolicyId =null, segments = null] 1 2018-03-10T11:23:04,504 | ERROR | opendaylight-cluster-data-akka.actor.default-dispatcher-2 | LocalThreePhaseCommitCohort | 230 - org.opendaylight.controller.sal-distributed-datastore - 1.7.0.SNAPSHOT | Failed to prepare transaction member-1-datastore-config-fe-0-txn-35013-0 on backend org.opendaylight.controller.md.sal.common.api.data.OptimisticLockFailedException: Optimistic lock failed. at org.opendaylight.controller.cluster.datastore.ShardDataTree.lambda$processNextPendingTransaction$0(ShardDataTree.java:740) ~[230:org.opendaylight.controller.sal-distributed-datastore:1.7.0.SNAPSHOT] at org.opendaylight.controller.cluster.datastore.ShardDataTree.processNextPending(ShardDataTree.java:778) [230:org.opendaylight.controller.sal-distributed-datastore:1.7.0.SNAPSHOT] at org.opendaylight.controller.cluster.datastore.ShardDataTree.processNextPendingTransaction(ShardDataTree.java:725) [230:org.opendaylight.controller.sal-distributed-datastore:1.7.0.SNAPSHOT] at org.opendaylight.controller.cluster.datastore.ShardDataTree.startCanCommit(ShardDataTree.java:808) [230:org.opendaylight.controller.sal-distributed-datastore:1.7.0.SNAPSHOT] at org.opendaylight.controller.cluster.datastore.SimpleShardDataTreeCohort.canCommit(SimpleShardDataTreeCohort.java:84) [230:org.opendaylight.controller.sal-distributed-datastore:1.7.0.SNAPSHOT] at org.opendaylight.controller.cluster.datastore.CohortEntry.canCommit(CohortEntry.java:97) [230:org.opendaylight.controller.sal-distributed-datastore:1.7.0.SNAPSHOT] at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator.handleCanCommit(ShardCommitCoordinator.java:236) [230:org.opendaylight.controller.sal-distributed-datastore:1.7.0.SNAPSHOT] at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator.handleReadyLocalTransaction(ShardCommitCoordinator.java:200) [230:org.opendaylight.controller.sal-distributed-datastore:1.7.0.SNAPSHOT] at org.opendaylight.controller.cluster.datastore.Shard.handleReadyLocalTransaction(Shard.java:731) [230:org.opendaylight.controller.sal-distributed-datastore:1.7.0.SNAPSHOT] at org.opendaylight.controller.cluster.datastore.Shard.handleNonRaftCommand(Shard.java:333) [230:org.opendaylight.controller.sal-distributed-datastore:1.7.0.SNAPSHOT] at org.opendaylight.controller.cluster.raft.RaftActor.handleCommand(RaftActor.java:270) [215:org.opendaylight.controller.sal-akka-raft:1.7.0.SNAPSHOT] at org.opendaylight.controller.cluster.common.actor.AbstractUntypedPersistentActor.onReceiveCommand(AbstractUntypedPersistentActor.java:44) [223:org.opendaylight.controller.sal-clustering-commons:1.7.0.SNAPSHOT] at akka.persistence.UntypedPersistentActor.onReceive(PersistentActor.scala:274) [43:com.typesafe.akka.persistence:2.5.4] at org.opendaylight.controller.cluster.common.actor.MeteringBehavior.apply(MeteringBehavior.java:104) [223:org.opendaylight.controller.sal-clustering-commons:1.7.0.SNAPSHOT] at akka.actor.ActorCell$$anonfun$become$1.applyOrElse(ActorCell.scala:545) [40:com.typesafe.akka.actor:2.5.4] at akka.actor.Actor.aroundReceive(Actor.scala:514) [40:com.typesafe.akka.actor:2.5.4] at akka.actor.Actor.aroundReceive$(Actor.scala:512) [40:com.typesafe.akka.actor:2.5.4] at akka.persistence.UntypedPersistentActor.akka$persistence$Eventsourced$$super$aroundReceive(PersistentActor.scala:272) [43:com.typesafe.akka.persistence:2.5.4] at akka.persistence.Eventsourced$$anon$1.stateReceive(Eventsourced.scala:663) [43:com.typesafe.akka.persistence:2.5.4] at akka.persistence.Eventsourced.aroundReceive(Eventsourced.scala:183) [43:com.typesafe.akka.persistence:2.5.4] at akka.persistence.Eventsourced.aroundReceive$(Eventsourced.scala:182) [43:com.typesafe.akka.persistence:2.5.4] at akka.persistence.UntypedPersistentActor.aroundReceive(PersistentActor.scala:272) [43:com.typesafe.akka.persistence:2.5.4] at akka.actor.ActorCell.receiveMessage(ActorCell.scala:527) [40:com.typesafe.akka.actor:2.5.4] at akka.actor.ActorCell.invoke(ActorCell.scala:496) [40:com.typesafe.akka.actor:2.5.4] at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257) [40:com.typesafe.akka.actor:2.5.4] at akka.dispatch.Mailbox.run(Mailbox.scala:224) [40:com.typesafe.akka.actor:2.5.4] at akka.dispatch.Mailbox.exec(Mailbox.scala:234) [40:com.typesafe.akka.actor:2.5.4] at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) [40:com.typesafe.akka.actor:2.5.4] at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) [40:com.typesafe.akka.actor:2.5.4] at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) [40:com.typesafe.akka.actor:2.5.4] at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) [40:com.typesafe.akka.actor:2.5.4] Caused by: org.opendaylight.yangtools.yang.data.api.schema.tree.ConflictingModificationAppliedException: Node was deleted by other transaction. at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkWriteApplicable(SchemaAwareApplyOperation.java:180) ~[427:org.opendaylight.yangtools.yang-data-impl:2.0.1] at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:135) ~[427:org.opendaylight.yangtools.yang-data-impl:2.0.1] at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkChildPreconditions(AbstractNodeContainerModificationStrategy.java:307) ~[427:org.opendaylight.yangtools.yang-data-impl:2.0.1] at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkMergeApplicable(AbstractNodeContainerModificationStrategy.java:315) ~[427:org.opendaylight.yangtools.yang-data-impl:2.0.1] at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:138) ~[427:org.opendaylight.yangtools.yang-data-impl:2.0.1] at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkChildPreconditions(AbstractNodeContainerModificationStrategy.java:307) ~[427:org.opendaylight.yangtools.yang-data-impl:2.0.1] at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkMergeApplicable(AbstractNodeContainerModificationStrategy.java:315) ~[427:org.opendaylight.yangtools.yang-data-impl:2.0.1] at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:138) ~[427:org.opendaylight.yangtools.yang-data-impl:2.0.1] at org.opendaylight.yangtools.yang.data.impl.schema.tree.StructuralContainerModificationStrategy.checkApplicable(StructuralContainerModificationStrategy.java:101) ~[427:org.opendaylight.yangtools.yang-data-impl:2.0.1] at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkChildPreconditions(AbstractNodeContainerModificationStrategy.java:307) ~[427:org.opendaylight.yangtools.yang-data-impl:2.0.1] at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkMergeApplicable(AbstractNodeContainerModificationStrategy.java:315) ~[427:org.opendaylight.yangtools.yang-data-impl:2.0.1] at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:138) ~[427:org.opendaylight.yangtools.yang-data-impl:2.0.1] at org.opendaylight.yangtools.yang.data.impl.schema.tree.StructuralContainerModificationStrategy.checkApplicable(StructuralContainerModificationStrategy.java:101) ~[427:org.opendaylight.yangtools.yang-data-impl:2.0.1] at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkChildPreconditions(AbstractNodeContainerModificationStrategy.java:307) ~[427:org.opendaylight.yangtools.yang-data-impl:2.0.1] at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkTouchApplicable(AbstractNodeContainerModificationStrategy.java:290) ~[427:org.opendaylight.yangtools.yang-data-impl:2.0.1] at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:132) ~[427:org.opendaylight.yangtools.yang-data-impl:2.0.1] at org.opendaylight.yangtools.yang.data.impl.schema.tree.RootModificationApplyOperation.checkApplicable(RootModificationApplyOperation.java:72) ~[427:org.opendaylight.yangtools.yang-data-impl:2.0.1] at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractDataTreeTip.validate(AbstractDataTreeTip.java:35) ~[427:org.opendaylight.yangtools.yang-data-impl:2.0.1] at org.opendaylight.controller.cluster.datastore.ShardDataTree.lambda$processNextPendingTransaction$0(ShardDataTree.java:732) ~[230:org.opendaylight.controller.sal-distributed-datastore:1.7.0.SNAPSHOT] ... 30 more</pre> </div></div> <p>This job had a variation of the exception:</p> <p><a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-openstack-queens-gate-stateful-fluorine/281/odl_1/odl1_karaf.log.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-openstack-queens-gate-stateful-fluorine/281/odl_1/odl1_karaf.log.gz</a></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>2018-04-30T13:13:28,143 | ERROR | qtp1114301045-1361 | AbstractTranscriberInterface | 390 - org.opendaylight.neutron.transcriber - 0.11.0.SNAPSHOT | Transaction failed java.util.concurrent.ExecutionException: OptimisticLockFailedException{message=Optimistic lock failed., errorList=[RpcError [message=Optimistic lock failed., severity=ERROR, errorType=APPLICATION, tag=resource-denied, applicationTag=null, info=null, cause=org.opendaylight.yangtools.yang.data.api.schema.tree.ConflictingModificationAppliedException: Node was deleted by other transaction.]]} at org.opendaylight.mdsal.common.api.MappingCheckedFuture.wrapInExecutionException(MappingCheckedFuture.java:66) [309:org.opendaylight.mdsal.common-api:2.5.0.SNAPSHOT] at org.opendaylight.mdsal.common.api.MappingCheckedFuture.get(MappingCheckedFuture.java:79) [309:org.opendaylight.mdsal.common-api:2.5.0.SNAPSHOT] at org.opendaylight.neutron.transcriber.AbstractTranscriberInterface.updateMd(AbstractTranscriberInterface.java:361) [390:org.opendaylight.neutron.transcriber:0.11.0.SNAPSHOT] </pre> </div></div> NEUTRON-157

Conflicting modification for path /(urn:opendaylight:neutron?revision=2015-07-12)neutron/networks/network/network[{(urn:opendaylight:neutron?revision=2015-07-12)uuid=b674297c-3ae3-4940-b3b8-149cb2da8161}

Bug High Resolved Done Michael Vorburger Sam Hague csit:exception Sat, 10 Mar 2018 14:18:05 +0000 Mon, 23 Jul 2018 10:20:53 +0000 Mon, 23 Jul 2018 10:20:53 +0000 Oxygen Fluorine Fluorine transcriber 0 3 <p>The test updates and then deletes a network in very quick succession. It seems that the order of events is such:</p> <ol> <li>client deletes network</li> <li>client updates that same network (yes, the rest call for the update comes <b>after</b> the delete</li> <li>ODL creates transaction for network delete</li> <li>ODL creates transaction for network update</li> <li>ODL submits transaction for network delete</li> <li>ODL submits transaction for network update which fails with the above exceptions.</li> </ol> <p>IMHO, this is not an ODL bug, especially because the delete is sent before the update that causes the exception. If this exception does not generally happen my guess is that's because the transaction created in (1) is committed before the transaction for the update is created. The client would still get a 404 but the exception would not appear in karaf.log.</p> <p>Just to clarify, when I say this is not an ODL bug I mean that ODL <b>behaved</b> in the correct manner. It would be better if the exception did not appear in the log, of course.</p> <p><tt>The CSIT run in the bug is 220. I checked and 218, 219, and 221 don't have the exception in karaf.log. All those runs except 219 have the REST calls for delete before the rest call for update. Here's the (non) pattern. </tt><br/> <tt>           218          219            220            221</tt></p> <p><tt>-----------------------------------------------------------</tt></p> <p><tt>bad order   X                           X              X</tt></p> <p><tt>exception                               X  </tt><tt> </tt></p> <p><tt>So the out-of-order rest calls seem to be a necessary, but not sufficient, cause of the OptimisticLockException. In order to get the exception the rest calls need to come out of order AND the update transaction must be created before the delete transaction is submitted.  </tt></p> <p>Here is another job with the exception:</p> <p><a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-openstack-queens-gate-stateful-oxygen/78/odl_1/odl1_karaf.log.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-openstack-queens-gate-stateful-oxygen/78/odl_1/odl1_karaf.log.gz</a></p> <p>Looks like the update coming after the delete may be a networking-odl journal issue. Here are some lines from the neutron logs for build 78:</p> <p>1) neutron executes an update on the network, line 108342:</p> <p>Mar 11 17:29:12 releng-45483-78-1-devstack-pike-0.vexxhost.net neutron-server<span class="error">&#91;19011&#93;</span>: INFO neutron.tests.unit.plugins.ml2.drivers.mechanism_logger <span class="error">&#91;None req-48ecbd9b-7166-4ee2-a70b-68512ba04bd4 tempest-BulkNetworkOpsTest-924910474 tempest-BulkNetworkOpsTest-924910474&#93;</span> update_network_precommit called with network settings {'provider:physical_network': None, 'ipv6_address_scope': None, 'revision_number': 3, 'port_security_enabled': True, 'provider:network_type': None, 'id': u'c828ffd6-bb52-4073-b738-f099316c2c31', 'router:external': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'shared': False, 'project_id': u'88fb4bf8685c433f91f2ecd6c8c76f30', 'status': u'ACTIVE', 'subnets': [], 'description': u'', 'tags': [], 'updated_at': '2018-03-11T17:29:12Z', 'provider:segmentation_id': None, 'name': u'tempest-network-1459092092', 'admin_state_up': True, 'tenant_id': u'88fb4bf8685c433f91f2ecd6c8c76f30', 'created_at': '2018-03-11T17:29:11Z', 'mtu': 1408, 'vlan_transparent': None} (original settings {'provider:physical_network': None, 'ipv6_address_scope': None, 'revision_number': 3, 'port_security_enabled': True, 'provider:network_type': None, 'id': u'c828ffd6-bb52-4073-b738-f099316c2c31', 'router:external': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'shared': False, 'project_id': u'88fb4bf8685c433f91f2ecd6c8c76f30', 'status': u'ACTIVE', 'subnets': [], 'description': u'', 'tags': [], 'updated_at': '2018-03-11T17:29:12Z', 'provider:segmentation_id': None, 'name': u'tempest-network-1459092092', 'admin_state_up': True, 'tenant_id': u'88fb4bf8685c433f91f2ecd6c8c76f30', 'created_at': '2018-03-11T17:29:11Z', 'mtu': 1408, 'vlan_transparent': None}) and network segments []</p> <p>2) REST call to neutron to delete the network, line 108487:</p> <p>Mar 11 17:29:12 releng-45483-78-1-devstack-pike-0.vexxhost.net neutron-server<span class="error">&#91;19011&#93;</span>: INFO neutron.wsgi <span class="error">&#91;None req-48ecbd9b-7166-4ee2-a70b-68512ba04bd4 tempest-BulkNetworkOpsTest-924910474 tempest-BulkNetworkOpsTest-924910474&#93;</span> 10.30.170.12 "DELETE /v2.0/networks/c828ffd6-bb52-4073-b738-f099316c2c31 HTTP/1.1" status: 204 len: 168 time: 0.8570170</p> <p>3) networking-odl REST calls odl to delete the network, line 108492:</p> <p>Mar 11 17:29:12 releng-45483-78-1-devstack-pike-0.vexxhost.net neutron-server<span class="error">&#91;19011&#93;</span>: DEBUG networking_odl.common.client <span class="error">&#91;None req-1afdbd9a-a31d-470e-9983-3259040dd939 None None&#93;</span> Sending METHOD (delete) URL (<a href="http://10.30.170.92:8181/controller/nb/v2/neutron/networks/c828ffd6-bb52-4073-b738-f099316c2c31" class="external-link" target="_blank" rel="nofollow noopener">http://10.30.170.92:8181/controller/nb/v2/neutron/networks/c828ffd6-bb52-4073-b738-f099316c2c31</a>) JSON (None) <tt>(pid=19600) request /opt/stack/networking-odl/networking_odl/common/client.py:89</tt></p> <p>4) networking odl REST calls odl to update the network, line 108509:</p> <p>Mar 11 17:29:13 releng-45483-78-1-devstack-pike-0.vexxhost.net neutron-server<span class="error">&#91;19011&#93;</span>: DEBUG networking_odl.common.client <span class="error">&#91;None req-8321e7b2-acc5-4860-8a13-8f988edbaf05 None None&#93;</span> Sending METHOD (put) URL (<a href="http://10.30.170.92:8181/controller/nb/v2/neutron/networks/c828ffd6-bb52-4073-b738-f099316c2c31" class="external-link" target="_blank" rel="nofollow noopener">http://10.30.170.92:8181/controller/nb/v2/neutron/networks/c828ffd6-bb52-4073-b738-f099316c2c31</a>) JSON ({</p> <p>Uploading the journalctl.log</p> <p>The new exception signature:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>2018-04-30T19:26:46,821 | ERROR | opendaylight-cluster-data-akka.actor.default-dispatcher-64 | LocalThreePhaseCommitCohort | 246 - org.opendaylight.controller.sal-distributed-datastore - 1.8.0.SNAPSHOT | Failed to prepare transaction member-1-datastore-config-fe-0-txn-39005-0 on backend org.opendaylight.mdsal.common.api.OptimisticLockFailedException: Optimistic lock failed. at org.opendaylight.controller.cluster.datastore.ShardDataTree.lambda$processNextPendingTransaction$0(ShardDataTree.java:740) ~[246:org.opendaylight.controller.sal-distributed-datastore:1.8.0.SNAPSHOT] at org.opendaylight.controller.cluster.datastore.ShardDataTree.processNextPending(ShardDataTree.java:778) [246:org.opendaylight.controller.sal-distributed-datastore:1.8.0.SNAPSHOT] at org.opendaylight.controller.cluster.datastore.ShardDataTree.processNextPendingTransaction(ShardDataTree.java:725) [246:org.opendaylight.controller.sal-distributed-datastore:1.8.0.SNAPSHOT] at org.opendaylight.controller.cluster.datastore.ShardDataTree.startCanCommit(ShardDataTree.java:808) [246:org.opendaylight.controller.sal-distributed-datastore:1.8.0.SNAPSHOT] at org.opendaylight.controller.cluster.datastore.SimpleShardDataTreeCohort.canCommit(SimpleShardDataTreeCohort.java:84) [246:org.opendaylight.controller.sal-distributed-datastore:1.8.0.SNAPSHOT] at org.opendaylight.controller.cluster.datastore.CohortEntry.canCommit(CohortEntry.java:97) [246:org.opendaylight.controller.sal-distributed-datastore:1.8.0.SNAPSHOT] at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator.handleCanCommit(ShardCommitCoordinator.java:236) [246:org.opendaylight.controller.sal-distributed-datastore:1.8.0.SNAPSHOT] at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator.handleReadyLocalTransaction(ShardCommitCoordinator.java:200) [246:org.opendaylight.controller.sal-distributed-datastore:1.8.0.SNAPSHOT] at org.opendaylight.controller.cluster.datastore.Shard.handleReadyLocalTransaction(Shard.java:731) [246:org.opendaylight.controller.sal-distributed-datastore:1.8.0.SNAPSHOT] at org.opendaylight.controller.cluster.datastore.Shard.handleNonRaftCommand(Shard.java:333) [246:org.opendaylight.controller.sal-distributed-datastore:1.8.0.SNAPSHOT] at org.opendaylight.controller.cluster.raft.RaftActor.handleCommand(RaftActor.java:270) [230:org.opendaylight.controller.sal-akka-raft:1.8.0.SNAPSHOT] at org.opendaylight.controller.cluster.common.actor.AbstractUntypedPersistentActor.onReceiveCommand(AbstractUntypedPersistentActor.java:44) [238:org.opendaylight.controller.sal-clustering-commons:1.8.0.SNAPSHOT] at akka.persistence.UntypedPersistentActor.onReceive(PersistentActor.scala:275) [43:com.typesafe.akka.persistence:2.5.11] at org.opendaylight.controller.cluster.common.actor.MeteringBehavior.apply(MeteringBehavior.java:104) [238:org.opendaylight.controller.sal-clustering-commons:1.8.0.SNAPSHOT] at akka.actor.ActorCell$$anonfun$become$1.applyOrElse(ActorCell.scala:608) [40:com.typesafe.akka.actor:2.5.11] at akka.actor.Actor.aroundReceive(Actor.scala:517) [40:com.typesafe.akka.actor:2.5.11] at akka.actor.Actor.aroundReceive$(Actor.scala:515) [40:com.typesafe.akka.actor:2.5.11] at akka.persistence.UntypedPersistentActor.akka$persistence$Eventsourced$$super$aroundReceive(PersistentActor.scala:273) [43:com.typesafe.akka.persistence:2.5.11] at akka.persistence.Eventsourced$$anon$1.stateReceive(Eventsourced.scala:691) [43:com.typesafe.akka.persistence:2.5.11] at akka.persistence.Eventsourced.aroundReceive(Eventsourced.scala:192) [43:com.typesafe.akka.persistence:2.5.11] at akka.persistence.Eventsourced.aroundReceive$(Eventsourced.scala:191) [43:com.typesafe.akka.persistence:2.5.11] at akka.persistence.UntypedPersistentActor.aroundReceive(PersistentActor.scala:273) [43:com.typesafe.akka.persistence:2.5.11] at akka.actor.ActorCell.receiveMessage(ActorCell.scala:590) [40:com.typesafe.akka.actor:2.5.11] at akka.actor.ActorCell.invoke(ActorCell.scala:559) [40:com.typesafe.akka.actor:2.5.11] at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257) [40:com.typesafe.akka.actor:2.5.11] at akka.dispatch.Mailbox.run(Mailbox.scala:224) [40:com.typesafe.akka.actor:2.5.11] at akka.dispatch.Mailbox.exec(Mailbox.scala:234) [40:com.typesafe.akka.actor:2.5.11] at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) [40:com.typesafe.akka.actor:2.5.11] at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) [40:com.typesafe.akka.actor:2.5.11] at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) [40:com.typesafe.akka.actor:2.5.11] at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) [40:com.typesafe.akka.actor:2.5.11] Caused by: org.opendaylight.yangtools.yang.data.api.schema.tree.ConflictingModificationAppliedException: Node was deleted by other transaction. at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkWriteApplicable(SchemaAwareApplyOperation.java:180) ~[446:org.opendaylight.yangtools.yang-data-impl:2.0.3] at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:135) ~[446:org.opendaylight.yangtools.yang-data-impl:2.0.3] at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkChildPreconditions(AbstractNodeContainerModificationStrategy.java:307) ~[446:org.opendaylight.yangtools.yang-data-impl:2.0.3] at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkMergeApplicable(AbstractNodeContainerModificationStrategy.java:315) ~[446:org.opendaylight.yangtools.yang-data-impl:2.0.3] at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:138) ~[446:org.opendaylight.yangtools.yang-data-impl:2.0.3] at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkChildPreconditions(AbstractNodeContainerModificationStrategy.java:307) ~[446:org.opendaylight.yangtools.yang-data-impl:2.0.3] at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkMergeApplicable(AbstractNodeContainerModificationStrategy.java:315) ~[446:org.opendaylight.yangtools.yang-data-impl:2.0.3] at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:138) ~[446:org.opendaylight.yangtools.yang-data-impl:2.0.3] at org.opendaylight.yangtools.yang.data.impl.schema.tree.StructuralContainerModificationStrategy.checkApplicable(StructuralContainerModificationStrategy.java:101) ~[446:org.opendaylight.yangtools.yang-data-impl:2.0.3] at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkChildPreconditions(AbstractNodeContainerModificationStrategy.java:307) ~[446:org.opendaylight.yangtools.yang-data-impl:2.0.3] at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkMergeApplicable(AbstractNodeContainerModificationStrategy.java:315) ~[446:org.opendaylight.yangtools.yang-data-impl:2.0.3] at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:138) ~[446:org.opendaylight.yangtools.yang-data-impl:2.0.3] at org.opendaylight.yangtools.yang.data.impl.schema.tree.StructuralContainerModificationStrategy.checkApplicable(StructuralContainerModificationStrategy.java:101) ~[446:org.opendaylight.yangtools.yang-data-impl:2.0.3] at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkChildPreconditions(AbstractNodeContainerModificationStrategy.java:307) ~[446:org.opendaylight.yangtools.yang-data-impl:2.0.3] at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkTouchApplicable(AbstractNodeContainerModificationStrategy.java:290) ~[446:org.opendaylight.yangtools.yang-data-impl:2.0.3] at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:132) ~[446:org.opendaylight.yangtools.yang-data-impl:2.0.3] at org.opendaylight.yangtools.yang.data.impl.schema.tree.RootModificationApplyOperation.checkApplicable(RootModificationApplyOperation.java:72) ~[446:org.opendaylight.yangtools.yang-data-impl:2.0.3] at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractDataTreeTip.validate(AbstractDataTreeTip.java:35) ~[446:org.opendaylight.yangtools.yang-data-impl:2.0.3] at org.opendaylight.controller.cluster.datastore.ShardDataTree.lambda$processNextPendingTransaction$0(ShardDataTree.java:732) ~[246:org.opendaylight.controller.sal-distributed-datastore:1.8.0.SNAPSHOT] ... 30 more 2018-04-30T19:26:46,887 | WARN | ForkJoinPool-1-worker-2 | NeutronPortChangeListener | 378 - org.opendaylight.netvirt.neutronvpn-impl - 0.7.0.SNAPSHOT | deleteOfPortInterface: Interface 9b3ee7ed-9c08-4bea-87df-94d41a6621d7 is not present 2018-04-30T19:26:46,826 | ERROR | qtp1760794530-1195 | AbstractTranscriberInterface | 391 - org.opendaylight.neutron.transcriber - 0.11.0.SNAPSHOT | Transaction failed java.util.concurrent.ExecutionException: OptimisticLockFailedException{message=Optimistic lock failed., errorList=[RpcError [message=Optimistic lock failed., severity=ERROR, errorType=APPLICATION, tag=resource-denied, applicationTag=null, info=null, cause=org.opendaylight.yangtools.yang.data.api.schema.tree.ConflictingModificationAppliedException: Node was deleted by other transaction.]]} at org.opendaylight.mdsal.common.api.MappingCheckedFuture.wrapInExecutionException(MappingCheckedFuture.java:66) [310:org.opendaylight.mdsal.common-api:2.5.0.SNAPSHOT] at org.opendaylight.mdsal.common.api.MappingCheckedFuture.get(MappingCheckedFuture.java:79) [310:org.opendaylight.mdsal.common-api:2.5.0.SNAPSHOT] at org.opendaylight.neutron.transcriber.AbstractTranscriberInterface.updateMd(AbstractTranscriberInterface.java:361) [391:org.opendaylight.neutron.transcriber:0.11.0.SNAPSHOT] at org.opendaylight.neutron.transcriber.AbstractTranscriberInterface.update(AbstractTranscriberInterface.java:551) [391:org.opendaylight.neutron.transcriber:0.11.0.SNAPSHOT] at org.opendaylight.neutron.transcriber.AbstractTranscriberInterface.update(AbstractTranscriberInterface.java:561) [391:org.opendaylight.neutron.transcriber:0.11.0.SNAPSHOT] at Proxy228f7842_f292_4ac6_b78a_c0331060da69.update(Unknown Source) [?:?] at Proxy6a4770d7_6ac6_423c_b657_f83c3dd02d03.update(Unknown Source) [?:?] at org.opendaylight.neutron.northbound.api.AbstractNeutronNorthbound.update(AbstractNeutronNorthbound.java:148) [389:org.opendaylight.neutron.northbound-api:0.11.0.SNAPSHOT] at org.opendaylight.neutron.northbound.api.NeutronNetworksNorthbound.updateNetwork(NeutronNetworksNorthbound.java:181) [389:org.opendaylight.neutron.northbound-api:0.11.0.SNAPSHOT] at sun.reflect.GeneratedMethodAccessor281.invoke(Unknown Source) ~[?:?] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:?] at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) [17:com.sun.jersey.jersey-server:1.19.4] at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205) [17:com.sun.jersey.jersey-server:1.19.4] at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75) [17:com.sun.jersey.jersey-server:1.19.4] at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302) [17:com.sun.jersey.jersey-server:1.19.4] at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) [17:com.sun.jersey.jersey-server:1.19.4] at com.sun.jersey.server.impl.uri.rules.ResourceObjectRule.accept(ResourceObjectRule.java:100) [17:com.sun.jersey.jersey-server:1.19.4] at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) [17:com.sun.jersey.jersey-server:1.19.4] at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84) [17:com.sun.jersey.jersey-server:1.19.4] at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542) [17:com.sun.jersey.jersey-server:1.19.4] at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473) [17:com.sun.jersey.jersey-server:1.19.4] at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419) [17:com.sun.jersey.jersey-server:1.19.4] at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409) [17:com.sun.jersey.jersey-server:1.19.4] at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409) [18:com.sun.jersey.servlet:1.19.4] at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558) [18:com.sun.jersey.servlet:1.19.4] at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733) [18:com.sun.jersey.servlet:1.19.4] at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) [70:javax.servlet-api:3.1.0] at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848) [161:org.eclipse.jetty.servlet:9.3.21.v20170918] at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1772) [161:org.eclipse.jetty.servlet:9.3.21.v20170918] at org.eclipse.jetty.servlets.CrossOriginFilter.handle(CrossOriginFilter.java:308) [162:org.eclipse.jetty.servlets:9.3.21.v20170918] at org.eclipse.jetty.servlets.CrossOriginFilter.doFilter(CrossOriginFilter.java:262) [162:org.eclipse.jetty.servlets:9.3.21.v20170918] at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759) [161:org.eclipse.jetty.servlet:9.3.21.v20170918] at org.eclipse.jetty.websocket.server.WebSocketUpgradeFilter.doFilter(WebSocketUpgradeFilter.java:205) [171:org.eclipse.jetty.websocket.server:9.3.21.v20170918] at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759) [161:org.eclipse.jetty.servlet:9.3.21.v20170918] at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:61) [142:org.apache.shiro.web:1.3.2] at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108) [142:org.apache.shiro.web:1.3.2] at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137) [142:org.apache.shiro.web:1.3.2] at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125) [142:org.apache.shiro.web:1.3.2] at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66) [142:org.apache.shiro.web:1.3.2] at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108) [142:org.apache.shiro.web:1.3.2] at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137) [142:org.apache.shiro.web:1.3.2] at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125) [142:org.apache.shiro.web:1.3.2] at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66) [142:org.apache.shiro.web:1.3.2] at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449) [142:org.apache.shiro.web:1.3.2] at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365) [142:org.apache.shiro.web:1.3.2] at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90) [141:org.apache.shiro.core:1.3.2] at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83) [141:org.apache.shiro.core:1.3.2] at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383) [141:org.apache.shiro.core:1.3.2] at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362) [142:org.apache.shiro.web:1.3.2] at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125) [142:org.apache.shiro.web:1.3.2] at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1751) [161:org.eclipse.jetty.servlet:9.3.21.v20170918] at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582) [161:org.eclipse.jetty.servlet:9.3.21.v20170918] at org.ops4j.pax.web.service.jetty.internal.HttpServiceServletHandler.doHandle(HttpServiceServletHandler.java:71) [463:org.ops4j.pax.web.pax-web-jetty:6.0.9] at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) [160:org.eclipse.jetty.server:9.3.21.v20170918] at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) [158:org.eclipse.jetty.security:9.3.21.v20170918] at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) [160:org.eclipse.jetty.server:9.3.21.v20170918] at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) [160:org.eclipse.jetty.server:9.3.21.v20170918] at org.ops4j.pax.web.service.jetty.internal.HttpServiceContext.doHandle(HttpServiceContext.java:284) [463:org.ops4j.pax.web.pax-web-jetty:6.0.9] at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512) [161:org.eclipse.jetty.servlet:9.3.21.v20170918] at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) [160:org.eclipse.jetty.server:9.3.21.v20170918] at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) [160:org.eclipse.jetty.server:9.3.21.v20170918] at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) [160:org.eclipse.jetty.server:9.3.21.v20170918] at org.ops4j.pax.web.service.jetty.internal.JettyServerHandlerCollection.handle(JettyServerHandlerCollection.java:80) [463:org.ops4j.pax.web.pax-web-jetty:6.0.9] at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) [160:org.eclipse.jetty.server:9.3.21.v20170918] at org.eclipse.jetty.server.Server.handle(Server.java:534) [160:org.eclipse.jetty.server:9.3.21.v20170918] at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333) [160:org.eclipse.jetty.server:9.3.21.v20170918] at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) [160:org.eclipse.jetty.server:9.3.21.v20170918] at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283) [152:org.eclipse.jetty.io:9.3.21.v20170918] at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108) [152:org.eclipse.jetty.io:9.3.21.v20170918] at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) [152:org.eclipse.jetty.io:9.3.21.v20170918] at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) [163:org.eclipse.jetty.util:9.3.21.v20170918] at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) [163:org.eclipse.jetty.util:9.3.21.v20170918] at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) [163:org.eclipse.jetty.util:9.3.21.v20170918] at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) [163:org.eclipse.jetty.util:9.3.21.v20170918] at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) [163:org.eclipse.jetty.util:9.3.21.v20170918] at java.lang.Thread.run(Thread.java:748) [?:?] Caused by: org.opendaylight.controller.md.sal.common.api.data.TransactionCommitFailedException: commit execution failed at org.opendaylight.controller.md.sal.dom.broker.impl.TransactionCommitFailedExceptionMapper.newWithCause(TransactionCommitFailedExceptionMapper.java:37) ~[?:?] at org.opendaylight.controller.md.sal.dom.broker.impl.TransactionCommitFailedExceptionMapper.newWithCause(TransactionCommitFailedExceptionMapper.java:19) ~[?:?] at org.opendaylight.yangtools.util.concurrent.ExceptionMapper.apply(ExceptionMapper.java:91) ~[?:?] at org.opendaylight.controller.md.sal.dom.broker.impl.TransactionCommitFailedExceptionMapper.apply(TransactionCommitFailedExceptionMapper.java:43) ~[?:?] at org.opendaylight.controller.md.sal.dom.broker.impl.TransactionCommitFailedExceptionMapper.apply(TransactionCommitFailedExceptionMapper.java:19) ~[?:?] at org.opendaylight.mdsal.common.api.MappingCheckedFuture.mapException(MappingCheckedFuture.java:62) ~[?:?] ... 77 more Caused by: org.opendaylight.mdsal.common.api.OptimisticLockFailedException: Optimistic lock failed. at org.opendaylight.controller.cluster.datastore.ShardDataTree.lambda$processNextPendingTransaction$0(ShardDataTree.java:740) ~[?:?] at org.opendaylight.controller.cluster.datastore.ShardDataTree.processNextPending(ShardDataTree.java:778) ~[?:?] at org.opendaylight.controller.cluster.datastore.ShardDataTree.processNextPendingTransaction(ShardDataTree.java:725) ~[?:?] at org.opendaylight.controller.cluster.datastore.ShardDataTree.startCanCommit(ShardDataTree.java:808) ~[?:?] at org.opendaylight.controller.cluster.datastore.SimpleShardDataTreeCohort.canCommit(SimpleShardDataTreeCohort.java:84) ~[?:?] at org.opendaylight.controller.cluster.datastore.CohortEntry.canCommit(CohortEntry.java:97) ~[?:?] at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator.handleCanCommit(ShardCommitCoordinator.java:236) ~[?:?] at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator.handleReadyLocalTransaction(ShardCommitCoordinator.java:200) ~[?:?] at org.opendaylight.controller.cluster.datastore.Shard.handleReadyLocalTransaction(Shard.java:731) ~[?:?] at org.opendaylight.controller.cluster.datastore.Shard.handleNonRaftCommand(Shard.java:333) ~[?:?] at org.opendaylight.controller.cluster.raft.RaftActor.handleCommand(RaftActor.java:270) ~[?:?] at org.opendaylight.controller.cluster.common.actor.AbstractUntypedPersistentActor.onReceiveCommand(AbstractUntypedPersistentActor.java:44) ~[?:?] at akka.persistence.UntypedPersistentActor.onReceive(PersistentActor.scala:275) ~[?:?] at org.opendaylight.controller.cluster.common.actor.MeteringBehavior.apply(MeteringBehavior.java:104) ~[?:?] at akka.actor.ActorCell$$anonfun$become$1.applyOrElse(ActorCell.scala:608) ~[?:?] at akka.actor.Actor.aroundReceive(Actor.scala:517) ~[?:?] at akka.actor.Actor.aroundReceive$(Actor.scala:515) ~[?:?] at akka.persistence.UntypedPersistentActor.akka$persistence$Eventsourced$$super$aroundReceive(PersistentActor.scala:273) ~[?:?] at akka.persistence.Eventsourced$$anon$1.stateReceive(Eventsourced.scala:691) ~[?:?] at akka.persistence.Eventsourced.aroundReceive(Eventsourced.scala:192) ~[?:?] at akka.persistence.Eventsourced.aroundReceive$(Eventsourced.scala:191) ~[?:?] at akka.persistence.UntypedPersistentActor.aroundReceive(PersistentActor.scala:273) ~[?:?] at akka.actor.ActorCell.receiveMessage(ActorCell.scala:590) ~[?:?] at akka.actor.ActorCell.invoke(ActorCell.scala:559) ~[?:?] at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257) ~[?:?] at akka.dispatch.Mailbox.run(Mailbox.scala:224) ~[?:?] at akka.dispatch.Mailbox.exec(Mailbox.scala:234) ~[?:?] at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) ~[?:?] at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) ~[?:?] at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) ~[?:?] at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) ~[?:?] Caused by: org.opendaylight.yangtools.yang.data.api.schema.tree.ConflictingModificationAppliedException: Node was deleted by other transaction. at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkWriteApplicable(SchemaAwareApplyOperation.java:180) ~[?:?] at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:135) ~[?:?] at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkChildPreconditions(AbstractNodeContainerModificationStrategy.java:307) ~[?:?] at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkMergeApplicable(AbstractNodeContainerModificationStrategy.java:315) ~[?:?] at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:138) ~[?:?] at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkChildPreconditions(AbstractNodeContainerModificationStrategy.java:307) ~[?:?] at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkMergeApplicable(AbstractNodeContainerModificationStrategy.java:315) ~[?:?] at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:138) ~[?:?] at org.opendaylight.yangtools.yang.data.impl.schema.tree.StructuralContainerModificationStrategy.checkApplicable(StructuralContainerModificationStrategy.java:101) ~[?:?] at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkChildPreconditions(AbstractNodeContainerModificationStrategy.java:307) ~[?:?] at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkMergeApplicable(AbstractNodeContainerModificationStrategy.java:315) ~[?:?] at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:138) ~[?:?] at org.opendaylight.yangtools.yang.data.impl.schema.tree.StructuralContainerModificationStrategy.checkApplicable(StructuralContainerModificationStrategy.java:101) ~[?:?] at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkChildPreconditions(AbstractNodeContainerModificationStrategy.java:307) ~[?:?] at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkTouchApplicable(AbstractNodeContainerModificationStrategy.java:290) ~[?:?] at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:132) ~[?:?] at org.opendaylight.yangtools.yang.data.impl.schema.tree.RootModificationApplyOperation.checkApplicable(RootModificationApplyOperation.java:72) ~[?:?] at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractDataTreeTip.validate(AbstractDataTreeTip.java:35) ~[?:?] at org.opendaylight.controller.cluster.datastore.ShardDataTree.lambda$processNextPendingTransaction$0(ShardDataTree.java:732) ~[?:?] at org.opendaylight.controller.cluster.datastore.ShardDataTree.processNextPending(ShardDataTree.java:778) ~[?:?] at org.opendaylight.controller.cluster.datastore.ShardDataTree.processNextPendingTransaction(ShardDataTree.java:725) ~[?:?] at org.opendaylight.controller.cluster.datastore.ShardDataTree.startCanCommit(ShardDataTree.java:808) ~[?:?] at org.opendaylight.controller.cluster.datastore.SimpleShardDataTreeCohort.canCommit(SimpleShardDataTreeCohort.java:84) ~[?:?] at org.opendaylight.controller.cluster.datastore.CohortEntry.canCommit(CohortEntry.java:97) ~[?:?] at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator.handleCanCommit(ShardCommitCoordinator.java:236) ~[?:?] at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator.handleReadyLocalTransaction(ShardCommitCoordinator.java:200) ~[?:?] at org.opendaylight.controller.cluster.datastore.Shard.handleReadyLocalTransaction(Shard.java:731) ~[?:?] at org.opendaylight.controller.cluster.datastore.Shard.handleNonRaftCommand(Shard.java:333) ~[?:?] at org.opendaylight.controller.cluster.raft.RaftActor.handleCommand(RaftActor.java:270) ~[?:?] at org.opendaylight.controller.cluster.common.actor.AbstractUntypedPersistentActor.onReceiveCommand(AbstractUntypedPersistentActor.java:44) ~[?:?] at akka.persistence.UntypedPersistentActor.onReceive(PersistentActor.scala:275) ~[?:?] at org.opendaylight.controller.cluster.common.actor.MeteringBehavior.apply(MeteringBehavior.java:104) ~[?:?] at akka.actor.ActorCell$$anonfun$become$1.applyOrElse(ActorCell.scala:608) ~[?:?] at akka.actor.Actor.aroundReceive(Actor.scala:517) ~[?:?] at akka.actor.Actor.aroundReceive$(Actor.scala:515) ~[?:?] at akka.persistence.UntypedPersistentActor.akka$persistence$Eventsourced$$super$aroundReceive(PersistentActor.scala:273) ~[?:?] at akka.persistence.Eventsourced$$anon$1.stateReceive(Eventsourced.scala:691) ~[?:?] at akka.persistence.Eventsourced.aroundReceive(Eventsourced.scala:192) ~[?:?] at akka.persistence.Eventsourced.aroundReceive$(Eventsourced.scala:191) ~[?:?] at akka.persistence.UntypedPersistentActor.aroundReceive(PersistentActor.scala:273) ~[?:?] at akka.actor.ActorCell.receiveMessage(ActorCell.scala:590) ~[?:?] at akka.actor.ActorCell.invoke(ActorCell.scala:559) ~[?:?] at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257) ~[?:?] at akka.dispatch.Mailbox.run(Mailbox.scala:224) ~[?:?] at akka.dispatch.Mailbox.exec(Mailbox.scala:234) ~[?:?] at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) ~[?:?] at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) ~[?:?] at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) ~[?:?] at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) ~[?:?] </pre> </div></div> <p>still seen: <a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-openstack-queens-upstream-stateful-oxygen/682/odl_1/odl1_karaf.log.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-openstack-queens-upstream-stateful-oxygen/682/odl_1/odl1_karaf.log.gz</a></p> <p>&gt; not an ODL bug I mean that ODL behaved in the correct manner.<br/> &gt; It would be better if the exception did not appear in the log, of course.</p> <p>I'm actively working on fixing this in ongoing <a href="https://git.opendaylight.org/gerrit/#/c/72735/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/72735/</a> WIP...</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=jhershbe" class="user-hover" rel="jhershbe">jhershbe</a> I don't want to "grab" this from you, but I'm assuming you're cool with it if I re-assign this one to me? <img class="emoticon" src="https://jira.opendaylight.org/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>question on the comment about not showing the exception - the bug is that networking-odl is sending events in the wrong order. ODL can't do anything with this since the required resources aren't there when the request is made. BUt, if we don't log this exception, how will we know if the root problem is fixed?</p> <p>&gt; the bug is that networking-odl is sending events in the wrong order. ODL can't do anything with this since the required resources aren't there when the request is made.</p> <p>well now that is of course the bigger more general problem we're after solving under <a href="https://jira.opendaylight.org/browse/NEUTRON-158" title="Dependency checking for Northbound API" class="issue-link" data-issue-key="NEUTRON-158">NEUTRON-158</a>... but aren't you mixing up 2 not directly related problems here? This bug is not (directly) about that, from what is written above, I thought. But if you think it is, then just make this Jira have a requires/depends on link to <a href="https://jira.opendaylight.org/browse/NEUTRON-158" title="Dependency checking for Northbound API" class="issue-link" data-issue-key="NEUTRON-158">NEUTRON-158</a> - fine for me.</p> <p>&gt; BUt, if we don't log this exception, how will we know if the root problem is fixed?</p> <p><a href="https://git.opendaylight.org/gerrit/#/c/72735/" class="external-link" target="_blank" rel="nofollow noopener">c/72735</a> makes Neutron actually return a HTTP 500 error in case of an OptimisticLockFailedException to the driver (instead of logging an error on ODL side but return 201 to the driver, which is the current behaviour, which IMHO is completely wrong), so the driver will log it as a failed operation - and, I guess (not sure), eventually retry.</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=shague" class="user-hover" rel="shague">shague</a> and <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=jhershbe" class="user-hover" rel="jhershbe">jhershbe</a> I was torn whether to really just send it to the Neutron drive to handle or log anyway, and more out of extra caution than because I think it's the right thing to do (actually I really think it's not...) at least WARN log it anyway in Neutron; so I'vehave proposed <a href="https://git.opendaylight.org/gerrit/#/c/73665/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/73665</a> (adds logging), with <a href="https://git.opendaylight.org/gerrit/#/c/73666/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/73666/</a> (supress warn log IFF OptimisticLockFailedException) as a possibly extension, for your review and discussion of that we really want to do here.</p> Blocks NETVIRT-1112 Duplicate NETVIRT-1112 NEUTRON-161 Development Rank 0|i03bin: