[NEUTRON-157] Conflicting modification for path /(urn:opendaylight:neutron?revision=2015-07-12)neutron/networks/network/network[{(urn:opendaylight:neutron?revision=2015-07-12)uuid=b674297c-3ae3-4940-b3b8-149cb2da8161} Created: 10/Mar/18  Updated: 23/Jul/18  Resolved: 23/Jul/18

Status: Resolved
Project: neutron
Component/s: transcriber
Affects Version/s: Oxygen, Fluorine
Fix Version/s: Fluorine

Type: Bug Priority: High
Reporter: Sam Hague Assignee: Michael Vorburger
Resolution: Done Votes: 0
Labels: csit:exception
Σ Remaining Estimate: Not Specified Remaining Estimate: Not Specified
Σ Time Spent: Not Specified Time Spent: Not Specified
Σ Original Estimate: Not Specified Original Estimate: Not Specified

Attachments: File journalctl.log.xz     File odl1_karaf.key.log.tar.xz    
Issue Links:
Blocks
blocks NETVIRT-1112 associateRouter NullPointerException ... Resolved
Duplicate
duplicates NETVIRT-1112 associateRouter NullPointerException ... Resolved
Sub-Tasks:
Key
Summary
Type
Status
Assignee
NEUTRON-161 revert CSIT patch when this bug is fixed Sub-task Resolved Jamo Luhrsen  

 Description   

https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-openstack-queens-upstream-stateful-oxygen/220/odl_1/odl1_karaf.log.gz

2018-03-10T11:23:04,503 | WARN  | opendaylight-cluster-data-shard-dispatcher-245 | ShardDataTree                    | 230 - org.opendaylight.controller.sal-distributed-datastore - 1.7.0.SNAPSHOT | member-1-shard-default-config: Store Tx member-1-datastore-config-fe-0-txn-35013-0: Conflicting modification for path /(urn:opendaylight:neutron?revision=2015-07-12)neutron/networks/network/network[{(urn:opendaylight:neutron?revision=2015-07-12)uuid=b674297c-3ae3-4940-b3b8-149cb2da8161}].
2018-03-10T11:23:04,504 | WARN  | qtp1247963896-1087 | AbstractTranscriberInterface     | 372 - org.opendaylight.neutron.transcriber - 0.10.0.SNAPSHOT | Got OptimisticLockFailedException - b674297c-3ae3-4940-b3b8-149cb2da8161 NeutronNetwork [networkUUID=b674297c-3ae3-4940-b3b8-149cb2da8161, networkName=tempest-BulkNetworkOpsTest-test-network-157138568, adminStateUp=true, shared=false, tenantID=7268bcf662294cdfa8fdfe71b9d47934, routerExternal=false, providerNetworkType=null, providerPhysicalNetwork=null, providerSegmentationID=null, status=null, qosPolicyId =null, segments = null] 1
2018-03-10T11:23:04,504 | ERROR | opendaylight-cluster-data-akka.actor.default-dispatcher-2 | LocalThreePhaseCommitCohort      | 230 - org.opendaylight.controller.sal-distributed-datastore - 1.7.0.SNAPSHOT | Failed to prepare transaction member-1-datastore-config-fe-0-txn-35013-0 on backend
org.opendaylight.controller.md.sal.common.api.data.OptimisticLockFailedException: Optimistic lock failed.
	at org.opendaylight.controller.cluster.datastore.ShardDataTree.lambda$processNextPendingTransaction$0(ShardDataTree.java:740) ~[230:org.opendaylight.controller.sal-distributed-datastore:1.7.0.SNAPSHOT]
	at org.opendaylight.controller.cluster.datastore.ShardDataTree.processNextPending(ShardDataTree.java:778) [230:org.opendaylight.controller.sal-distributed-datastore:1.7.0.SNAPSHOT]
	at org.opendaylight.controller.cluster.datastore.ShardDataTree.processNextPendingTransaction(ShardDataTree.java:725) [230:org.opendaylight.controller.sal-distributed-datastore:1.7.0.SNAPSHOT]
	at org.opendaylight.controller.cluster.datastore.ShardDataTree.startCanCommit(ShardDataTree.java:808) [230:org.opendaylight.controller.sal-distributed-datastore:1.7.0.SNAPSHOT]
	at org.opendaylight.controller.cluster.datastore.SimpleShardDataTreeCohort.canCommit(SimpleShardDataTreeCohort.java:84) [230:org.opendaylight.controller.sal-distributed-datastore:1.7.0.SNAPSHOT]
	at org.opendaylight.controller.cluster.datastore.CohortEntry.canCommit(CohortEntry.java:97) [230:org.opendaylight.controller.sal-distributed-datastore:1.7.0.SNAPSHOT]
	at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator.handleCanCommit(ShardCommitCoordinator.java:236) [230:org.opendaylight.controller.sal-distributed-datastore:1.7.0.SNAPSHOT]
	at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator.handleReadyLocalTransaction(ShardCommitCoordinator.java:200) [230:org.opendaylight.controller.sal-distributed-datastore:1.7.0.SNAPSHOT]
	at org.opendaylight.controller.cluster.datastore.Shard.handleReadyLocalTransaction(Shard.java:731) [230:org.opendaylight.controller.sal-distributed-datastore:1.7.0.SNAPSHOT]
	at org.opendaylight.controller.cluster.datastore.Shard.handleNonRaftCommand(Shard.java:333) [230:org.opendaylight.controller.sal-distributed-datastore:1.7.0.SNAPSHOT]
	at org.opendaylight.controller.cluster.raft.RaftActor.handleCommand(RaftActor.java:270) [215:org.opendaylight.controller.sal-akka-raft:1.7.0.SNAPSHOT]
	at org.opendaylight.controller.cluster.common.actor.AbstractUntypedPersistentActor.onReceiveCommand(AbstractUntypedPersistentActor.java:44) [223:org.opendaylight.controller.sal-clustering-commons:1.7.0.SNAPSHOT]
	at akka.persistence.UntypedPersistentActor.onReceive(PersistentActor.scala:274) [43:com.typesafe.akka.persistence:2.5.4]
	at org.opendaylight.controller.cluster.common.actor.MeteringBehavior.apply(MeteringBehavior.java:104) [223:org.opendaylight.controller.sal-clustering-commons:1.7.0.SNAPSHOT]
	at akka.actor.ActorCell$$anonfun$become$1.applyOrElse(ActorCell.scala:545) [40:com.typesafe.akka.actor:2.5.4]
	at akka.actor.Actor.aroundReceive(Actor.scala:514) [40:com.typesafe.akka.actor:2.5.4]
	at akka.actor.Actor.aroundReceive$(Actor.scala:512) [40:com.typesafe.akka.actor:2.5.4]
	at akka.persistence.UntypedPersistentActor.akka$persistence$Eventsourced$$super$aroundReceive(PersistentActor.scala:272) [43:com.typesafe.akka.persistence:2.5.4]
	at akka.persistence.Eventsourced$$anon$1.stateReceive(Eventsourced.scala:663) [43:com.typesafe.akka.persistence:2.5.4]
	at akka.persistence.Eventsourced.aroundReceive(Eventsourced.scala:183) [43:com.typesafe.akka.persistence:2.5.4]
	at akka.persistence.Eventsourced.aroundReceive$(Eventsourced.scala:182) [43:com.typesafe.akka.persistence:2.5.4]
	at akka.persistence.UntypedPersistentActor.aroundReceive(PersistentActor.scala:272) [43:com.typesafe.akka.persistence:2.5.4]
	at akka.actor.ActorCell.receiveMessage(ActorCell.scala:527) [40:com.typesafe.akka.actor:2.5.4]
	at akka.actor.ActorCell.invoke(ActorCell.scala:496) [40:com.typesafe.akka.actor:2.5.4]
	at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257) [40:com.typesafe.akka.actor:2.5.4]
	at akka.dispatch.Mailbox.run(Mailbox.scala:224) [40:com.typesafe.akka.actor:2.5.4]
	at akka.dispatch.Mailbox.exec(Mailbox.scala:234) [40:com.typesafe.akka.actor:2.5.4]
	at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) [40:com.typesafe.akka.actor:2.5.4]
	at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) [40:com.typesafe.akka.actor:2.5.4]
	at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) [40:com.typesafe.akka.actor:2.5.4]
	at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) [40:com.typesafe.akka.actor:2.5.4]
Caused by: org.opendaylight.yangtools.yang.data.api.schema.tree.ConflictingModificationAppliedException: Node was deleted by other transaction.
	at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkWriteApplicable(SchemaAwareApplyOperation.java:180) ~[427:org.opendaylight.yangtools.yang-data-impl:2.0.1]
	at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:135) ~[427:org.opendaylight.yangtools.yang-data-impl:2.0.1]
	at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkChildPreconditions(AbstractNodeContainerModificationStrategy.java:307) ~[427:org.opendaylight.yangtools.yang-data-impl:2.0.1]
	at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkMergeApplicable(AbstractNodeContainerModificationStrategy.java:315) ~[427:org.opendaylight.yangtools.yang-data-impl:2.0.1]
	at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:138) ~[427:org.opendaylight.yangtools.yang-data-impl:2.0.1]
	at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkChildPreconditions(AbstractNodeContainerModificationStrategy.java:307) ~[427:org.opendaylight.yangtools.yang-data-impl:2.0.1]
	at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkMergeApplicable(AbstractNodeContainerModificationStrategy.java:315) ~[427:org.opendaylight.yangtools.yang-data-impl:2.0.1]
	at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:138) ~[427:org.opendaylight.yangtools.yang-data-impl:2.0.1]
	at org.opendaylight.yangtools.yang.data.impl.schema.tree.StructuralContainerModificationStrategy.checkApplicable(StructuralContainerModificationStrategy.java:101) ~[427:org.opendaylight.yangtools.yang-data-impl:2.0.1]
	at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkChildPreconditions(AbstractNodeContainerModificationStrategy.java:307) ~[427:org.opendaylight.yangtools.yang-data-impl:2.0.1]
	at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkMergeApplicable(AbstractNodeContainerModificationStrategy.java:315) ~[427:org.opendaylight.yangtools.yang-data-impl:2.0.1]
	at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:138) ~[427:org.opendaylight.yangtools.yang-data-impl:2.0.1]
	at org.opendaylight.yangtools.yang.data.impl.schema.tree.StructuralContainerModificationStrategy.checkApplicable(StructuralContainerModificationStrategy.java:101) ~[427:org.opendaylight.yangtools.yang-data-impl:2.0.1]
	at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkChildPreconditions(AbstractNodeContainerModificationStrategy.java:307) ~[427:org.opendaylight.yangtools.yang-data-impl:2.0.1]
	at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkTouchApplicable(AbstractNodeContainerModificationStrategy.java:290) ~[427:org.opendaylight.yangtools.yang-data-impl:2.0.1]
	at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:132) ~[427:org.opendaylight.yangtools.yang-data-impl:2.0.1]
	at org.opendaylight.yangtools.yang.data.impl.schema.tree.RootModificationApplyOperation.checkApplicable(RootModificationApplyOperation.java:72) ~[427:org.opendaylight.yangtools.yang-data-impl:2.0.1]
	at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractDataTreeTip.validate(AbstractDataTreeTip.java:35) ~[427:org.opendaylight.yangtools.yang-data-impl:2.0.1]
	at org.opendaylight.controller.cluster.datastore.ShardDataTree.lambda$processNextPendingTransaction$0(ShardDataTree.java:732) ~[230:org.opendaylight.controller.sal-distributed-datastore:1.7.0.SNAPSHOT]
	... 30 more

This job had a variation of the exception:

https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-openstack-queens-gate-stateful-fluorine/281/odl_1/odl1_karaf.log.gz

2018-04-30T13:13:28,143 | ERROR | qtp1114301045-1361 | AbstractTranscriberInterface     | 390 - org.opendaylight.neutron.transcriber - 0.11.0.SNAPSHOT | Transaction failed
java.util.concurrent.ExecutionException: OptimisticLockFailedException{message=Optimistic lock failed., errorList=[RpcError [message=Optimistic lock failed., severity=ERROR, errorType=APPLICATION, tag=resource-denied, applicationTag=null, info=null, cause=org.opendaylight.yangtools.yang.data.api.schema.tree.ConflictingModificationAppliedException: Node was deleted by other transaction.]]}
	at org.opendaylight.mdsal.common.api.MappingCheckedFuture.wrapInExecutionException(MappingCheckedFuture.java:66) [309:org.opendaylight.mdsal.common-api:2.5.0.SNAPSHOT]
	at org.opendaylight.mdsal.common.api.MappingCheckedFuture.get(MappingCheckedFuture.java:79) [309:org.opendaylight.mdsal.common-api:2.5.0.SNAPSHOT]
	at org.opendaylight.neutron.transcriber.AbstractTranscriberInterface.updateMd(AbstractTranscriberInterface.java:361) [390:org.opendaylight.neutron.transcriber:0.11.0.SNAPSHOT]


 Comments   
Comment by Josh Hershberg [ 11/Mar/18 ]

The test updates and then deletes a network in very quick succession. It seems that the order of events is such:

  1. client deletes network
  2. client updates that same network (yes, the rest call for the update comes after the delete
  3. ODL creates transaction for network delete
  4. ODL creates transaction for network update
  5. ODL submits transaction for network delete
  6. ODL submits transaction for network update which fails with the above exceptions.

IMHO, this is not an ODL bug, especially because the delete is sent before the update that causes the exception. If this exception does not generally happen my guess is that's because the transaction created in (1) is committed before the transaction for the update is created. The client would still get a 404 but the exception would not appear in karaf.log.

Comment by Josh Hershberg [ 11/Mar/18 ]

Just to clarify, when I say this is not an ODL bug I mean that ODL behaved in the correct manner. It would be better if the exception did not appear in the log, of course.

Comment by Josh Hershberg [ 11/Mar/18 ]

The CSIT run in the bug is 220. I checked and 218, 219, and 221 don't have the exception in karaf.log. All those runs except 219 have the REST calls for delete before the rest call for update. Here's the (non) pattern. 
           218          219            220            221

-----------------------------------------------------------

bad order   X                           X              X

exception                               X   

So the out-of-order rest calls seem to be a necessary, but not sufficient, cause of the OptimisticLockException. In order to get the exception the rest calls need to come out of order AND the update transaction must be created before the delete transaction is submitted.  

Comment by Sam Hague [ 11/Mar/18 ]

Here is another job with the exception:

https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-openstack-queens-gate-stateful-oxygen/78/odl_1/odl1_karaf.log.gz

Comment by Josh Hershberg [ 12/Mar/18 ]

Looks like the update coming after the delete may be a networking-odl journal issue. Here are some lines from the neutron logs for build 78:

1) neutron executes an update on the network, line 108342:

Mar 11 17:29:12 releng-45483-78-1-devstack-pike-0.vexxhost.net neutron-server[19011]: INFO neutron.tests.unit.plugins.ml2.drivers.mechanism_logger [None req-48ecbd9b-7166-4ee2-a70b-68512ba04bd4 tempest-BulkNetworkOpsTest-924910474 tempest-BulkNetworkOpsTest-924910474] update_network_precommit called with network settings {'provider:physical_network': None, 'ipv6_address_scope': None, 'revision_number': 3, 'port_security_enabled': True, 'provider:network_type': None, 'id': u'c828ffd6-bb52-4073-b738-f099316c2c31', 'router:external': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'shared': False, 'project_id': u'88fb4bf8685c433f91f2ecd6c8c76f30', 'status': u'ACTIVE', 'subnets': [], 'description': u'', 'tags': [], 'updated_at': '2018-03-11T17:29:12Z', 'provider:segmentation_id': None, 'name': u'tempest-network-1459092092', 'admin_state_up': True, 'tenant_id': u'88fb4bf8685c433f91f2ecd6c8c76f30', 'created_at': '2018-03-11T17:29:11Z', 'mtu': 1408, 'vlan_transparent': None} (original settings {'provider:physical_network': None, 'ipv6_address_scope': None, 'revision_number': 3, 'port_security_enabled': True, 'provider:network_type': None, 'id': u'c828ffd6-bb52-4073-b738-f099316c2c31', 'router:external': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'shared': False, 'project_id': u'88fb4bf8685c433f91f2ecd6c8c76f30', 'status': u'ACTIVE', 'subnets': [], 'description': u'', 'tags': [], 'updated_at': '2018-03-11T17:29:12Z', 'provider:segmentation_id': None, 'name': u'tempest-network-1459092092', 'admin_state_up': True, 'tenant_id': u'88fb4bf8685c433f91f2ecd6c8c76f30', 'created_at': '2018-03-11T17:29:11Z', 'mtu': 1408, 'vlan_transparent': None}) and network segments []

2) REST call to neutron to delete the network, line 108487:

Mar 11 17:29:12 releng-45483-78-1-devstack-pike-0.vexxhost.net neutron-server[19011]: INFO neutron.wsgi [None req-48ecbd9b-7166-4ee2-a70b-68512ba04bd4 tempest-BulkNetworkOpsTest-924910474 tempest-BulkNetworkOpsTest-924910474] 10.30.170.12 "DELETE /v2.0/networks/c828ffd6-bb52-4073-b738-f099316c2c31 HTTP/1.1" status: 204 len: 168 time: 0.8570170

3) networking-odl REST calls odl to delete the network, line 108492:

Mar 11 17:29:12 releng-45483-78-1-devstack-pike-0.vexxhost.net neutron-server[19011]: DEBUG networking_odl.common.client [None req-1afdbd9a-a31d-470e-9983-3259040dd939 None None] Sending METHOD (delete) URL (http://10.30.170.92:8181/controller/nb/v2/neutron/networks/c828ffd6-bb52-4073-b738-f099316c2c31) JSON (None) (pid=19600) request /opt/stack/networking-odl/networking_odl/common/client.py:89

4) networking odl REST calls odl to update the network, line 108509:

Mar 11 17:29:13 releng-45483-78-1-devstack-pike-0.vexxhost.net neutron-server[19011]: DEBUG networking_odl.common.client [None req-8321e7b2-acc5-4860-8a13-8f988edbaf05 None None] Sending METHOD (put) URL (http://10.30.170.92:8181/controller/nb/v2/neutron/networks/c828ffd6-bb52-4073-b738-f099316c2c31) JSON ({

Uploading the journalctl.log

Comment by Sam Hague [ 01/May/18 ]

The new exception signature:

2018-04-30T19:26:46,821 | ERROR | opendaylight-cluster-data-akka.actor.default-dispatcher-64 | LocalThreePhaseCommitCohort      | 246 - org.opendaylight.controller.sal-distributed-datastore - 1.8.0.SNAPSHOT | Failed to prepare transaction member-1-datastore-config-fe-0-txn-39005-0 on backend
org.opendaylight.mdsal.common.api.OptimisticLockFailedException: Optimistic lock failed.
	at org.opendaylight.controller.cluster.datastore.ShardDataTree.lambda$processNextPendingTransaction$0(ShardDataTree.java:740) ~[246:org.opendaylight.controller.sal-distributed-datastore:1.8.0.SNAPSHOT]
	at org.opendaylight.controller.cluster.datastore.ShardDataTree.processNextPending(ShardDataTree.java:778) [246:org.opendaylight.controller.sal-distributed-datastore:1.8.0.SNAPSHOT]
	at org.opendaylight.controller.cluster.datastore.ShardDataTree.processNextPendingTransaction(ShardDataTree.java:725) [246:org.opendaylight.controller.sal-distributed-datastore:1.8.0.SNAPSHOT]
	at org.opendaylight.controller.cluster.datastore.ShardDataTree.startCanCommit(ShardDataTree.java:808) [246:org.opendaylight.controller.sal-distributed-datastore:1.8.0.SNAPSHOT]
	at org.opendaylight.controller.cluster.datastore.SimpleShardDataTreeCohort.canCommit(SimpleShardDataTreeCohort.java:84) [246:org.opendaylight.controller.sal-distributed-datastore:1.8.0.SNAPSHOT]
	at org.opendaylight.controller.cluster.datastore.CohortEntry.canCommit(CohortEntry.java:97) [246:org.opendaylight.controller.sal-distributed-datastore:1.8.0.SNAPSHOT]
	at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator.handleCanCommit(ShardCommitCoordinator.java:236) [246:org.opendaylight.controller.sal-distributed-datastore:1.8.0.SNAPSHOT]
	at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator.handleReadyLocalTransaction(ShardCommitCoordinator.java:200) [246:org.opendaylight.controller.sal-distributed-datastore:1.8.0.SNAPSHOT]
	at org.opendaylight.controller.cluster.datastore.Shard.handleReadyLocalTransaction(Shard.java:731) [246:org.opendaylight.controller.sal-distributed-datastore:1.8.0.SNAPSHOT]
	at org.opendaylight.controller.cluster.datastore.Shard.handleNonRaftCommand(Shard.java:333) [246:org.opendaylight.controller.sal-distributed-datastore:1.8.0.SNAPSHOT]
	at org.opendaylight.controller.cluster.raft.RaftActor.handleCommand(RaftActor.java:270) [230:org.opendaylight.controller.sal-akka-raft:1.8.0.SNAPSHOT]
	at org.opendaylight.controller.cluster.common.actor.AbstractUntypedPersistentActor.onReceiveCommand(AbstractUntypedPersistentActor.java:44) [238:org.opendaylight.controller.sal-clustering-commons:1.8.0.SNAPSHOT]
	at akka.persistence.UntypedPersistentActor.onReceive(PersistentActor.scala:275) [43:com.typesafe.akka.persistence:2.5.11]
	at org.opendaylight.controller.cluster.common.actor.MeteringBehavior.apply(MeteringBehavior.java:104) [238:org.opendaylight.controller.sal-clustering-commons:1.8.0.SNAPSHOT]
	at akka.actor.ActorCell$$anonfun$become$1.applyOrElse(ActorCell.scala:608) [40:com.typesafe.akka.actor:2.5.11]
	at akka.actor.Actor.aroundReceive(Actor.scala:517) [40:com.typesafe.akka.actor:2.5.11]
	at akka.actor.Actor.aroundReceive$(Actor.scala:515) [40:com.typesafe.akka.actor:2.5.11]
	at akka.persistence.UntypedPersistentActor.akka$persistence$Eventsourced$$super$aroundReceive(PersistentActor.scala:273) [43:com.typesafe.akka.persistence:2.5.11]
	at akka.persistence.Eventsourced$$anon$1.stateReceive(Eventsourced.scala:691) [43:com.typesafe.akka.persistence:2.5.11]
	at akka.persistence.Eventsourced.aroundReceive(Eventsourced.scala:192) [43:com.typesafe.akka.persistence:2.5.11]
	at akka.persistence.Eventsourced.aroundReceive$(Eventsourced.scala:191) [43:com.typesafe.akka.persistence:2.5.11]
	at akka.persistence.UntypedPersistentActor.aroundReceive(PersistentActor.scala:273) [43:com.typesafe.akka.persistence:2.5.11]
	at akka.actor.ActorCell.receiveMessage(ActorCell.scala:590) [40:com.typesafe.akka.actor:2.5.11]
	at akka.actor.ActorCell.invoke(ActorCell.scala:559) [40:com.typesafe.akka.actor:2.5.11]
	at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257) [40:com.typesafe.akka.actor:2.5.11]
	at akka.dispatch.Mailbox.run(Mailbox.scala:224) [40:com.typesafe.akka.actor:2.5.11]
	at akka.dispatch.Mailbox.exec(Mailbox.scala:234) [40:com.typesafe.akka.actor:2.5.11]
	at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) [40:com.typesafe.akka.actor:2.5.11]
	at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) [40:com.typesafe.akka.actor:2.5.11]
	at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) [40:com.typesafe.akka.actor:2.5.11]
	at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) [40:com.typesafe.akka.actor:2.5.11]
Caused by: org.opendaylight.yangtools.yang.data.api.schema.tree.ConflictingModificationAppliedException: Node was deleted by other transaction.
	at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkWriteApplicable(SchemaAwareApplyOperation.java:180) ~[446:org.opendaylight.yangtools.yang-data-impl:2.0.3]
	at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:135) ~[446:org.opendaylight.yangtools.yang-data-impl:2.0.3]
	at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkChildPreconditions(AbstractNodeContainerModificationStrategy.java:307) ~[446:org.opendaylight.yangtools.yang-data-impl:2.0.3]
	at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkMergeApplicable(AbstractNodeContainerModificationStrategy.java:315) ~[446:org.opendaylight.yangtools.yang-data-impl:2.0.3]
	at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:138) ~[446:org.opendaylight.yangtools.yang-data-impl:2.0.3]
	at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkChildPreconditions(AbstractNodeContainerModificationStrategy.java:307) ~[446:org.opendaylight.yangtools.yang-data-impl:2.0.3]
	at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkMergeApplicable(AbstractNodeContainerModificationStrategy.java:315) ~[446:org.opendaylight.yangtools.yang-data-impl:2.0.3]
	at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:138) ~[446:org.opendaylight.yangtools.yang-data-impl:2.0.3]
	at org.opendaylight.yangtools.yang.data.impl.schema.tree.StructuralContainerModificationStrategy.checkApplicable(StructuralContainerModificationStrategy.java:101) ~[446:org.opendaylight.yangtools.yang-data-impl:2.0.3]
	at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkChildPreconditions(AbstractNodeContainerModificationStrategy.java:307) ~[446:org.opendaylight.yangtools.yang-data-impl:2.0.3]
	at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkMergeApplicable(AbstractNodeContainerModificationStrategy.java:315) ~[446:org.opendaylight.yangtools.yang-data-impl:2.0.3]
	at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:138) ~[446:org.opendaylight.yangtools.yang-data-impl:2.0.3]
	at org.opendaylight.yangtools.yang.data.impl.schema.tree.StructuralContainerModificationStrategy.checkApplicable(StructuralContainerModificationStrategy.java:101) ~[446:org.opendaylight.yangtools.yang-data-impl:2.0.3]
	at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkChildPreconditions(AbstractNodeContainerModificationStrategy.java:307) ~[446:org.opendaylight.yangtools.yang-data-impl:2.0.3]
	at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkTouchApplicable(AbstractNodeContainerModificationStrategy.java:290) ~[446:org.opendaylight.yangtools.yang-data-impl:2.0.3]
	at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:132) ~[446:org.opendaylight.yangtools.yang-data-impl:2.0.3]
	at org.opendaylight.yangtools.yang.data.impl.schema.tree.RootModificationApplyOperation.checkApplicable(RootModificationApplyOperation.java:72) ~[446:org.opendaylight.yangtools.yang-data-impl:2.0.3]
	at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractDataTreeTip.validate(AbstractDataTreeTip.java:35) ~[446:org.opendaylight.yangtools.yang-data-impl:2.0.3]
	at org.opendaylight.controller.cluster.datastore.ShardDataTree.lambda$processNextPendingTransaction$0(ShardDataTree.java:732) ~[246:org.opendaylight.controller.sal-distributed-datastore:1.8.0.SNAPSHOT]
	... 30 more
2018-04-30T19:26:46,887 | WARN  | ForkJoinPool-1-worker-2 | NeutronPortChangeListener        | 378 - org.opendaylight.netvirt.neutronvpn-impl - 0.7.0.SNAPSHOT | deleteOfPortInterface: Interface 9b3ee7ed-9c08-4bea-87df-94d41a6621d7 is not present
2018-04-30T19:26:46,826 | ERROR | qtp1760794530-1195 | AbstractTranscriberInterface     | 391 - org.opendaylight.neutron.transcriber - 0.11.0.SNAPSHOT | Transaction failed
java.util.concurrent.ExecutionException: OptimisticLockFailedException{message=Optimistic lock failed., errorList=[RpcError [message=Optimistic lock failed., severity=ERROR, errorType=APPLICATION, tag=resource-denied, applicationTag=null, info=null, cause=org.opendaylight.yangtools.yang.data.api.schema.tree.ConflictingModificationAppliedException: Node was deleted by other transaction.]]}
	at org.opendaylight.mdsal.common.api.MappingCheckedFuture.wrapInExecutionException(MappingCheckedFuture.java:66) [310:org.opendaylight.mdsal.common-api:2.5.0.SNAPSHOT]
	at org.opendaylight.mdsal.common.api.MappingCheckedFuture.get(MappingCheckedFuture.java:79) [310:org.opendaylight.mdsal.common-api:2.5.0.SNAPSHOT]
	at org.opendaylight.neutron.transcriber.AbstractTranscriberInterface.updateMd(AbstractTranscriberInterface.java:361) [391:org.opendaylight.neutron.transcriber:0.11.0.SNAPSHOT]
	at org.opendaylight.neutron.transcriber.AbstractTranscriberInterface.update(AbstractTranscriberInterface.java:551) [391:org.opendaylight.neutron.transcriber:0.11.0.SNAPSHOT]
	at org.opendaylight.neutron.transcriber.AbstractTranscriberInterface.update(AbstractTranscriberInterface.java:561) [391:org.opendaylight.neutron.transcriber:0.11.0.SNAPSHOT]
	at Proxy228f7842_f292_4ac6_b78a_c0331060da69.update(Unknown Source) [?:?]
	at Proxy6a4770d7_6ac6_423c_b657_f83c3dd02d03.update(Unknown Source) [?:?]
	at org.opendaylight.neutron.northbound.api.AbstractNeutronNorthbound.update(AbstractNeutronNorthbound.java:148) [389:org.opendaylight.neutron.northbound-api:0.11.0.SNAPSHOT]
	at org.opendaylight.neutron.northbound.api.NeutronNetworksNorthbound.updateNetwork(NeutronNetworksNorthbound.java:181) [389:org.opendaylight.neutron.northbound-api:0.11.0.SNAPSHOT]
	at sun.reflect.GeneratedMethodAccessor281.invoke(Unknown Source) ~[?:?]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[?:?]
	at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) [17:com.sun.jersey.jersey-server:1.19.4]
	at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205) [17:com.sun.jersey.jersey-server:1.19.4]
	at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75) [17:com.sun.jersey.jersey-server:1.19.4]
	at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302) [17:com.sun.jersey.jersey-server:1.19.4]
	at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) [17:com.sun.jersey.jersey-server:1.19.4]
	at com.sun.jersey.server.impl.uri.rules.ResourceObjectRule.accept(ResourceObjectRule.java:100) [17:com.sun.jersey.jersey-server:1.19.4]
	at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) [17:com.sun.jersey.jersey-server:1.19.4]
	at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84) [17:com.sun.jersey.jersey-server:1.19.4]
	at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542) [17:com.sun.jersey.jersey-server:1.19.4]
	at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473) [17:com.sun.jersey.jersey-server:1.19.4]
	at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419) [17:com.sun.jersey.jersey-server:1.19.4]
	at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409) [17:com.sun.jersey.jersey-server:1.19.4]
	at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409) [18:com.sun.jersey.servlet:1.19.4]
	at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558) [18:com.sun.jersey.servlet:1.19.4]
	at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733) [18:com.sun.jersey.servlet:1.19.4]
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) [70:javax.servlet-api:3.1.0]
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848) [161:org.eclipse.jetty.servlet:9.3.21.v20170918]
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1772) [161:org.eclipse.jetty.servlet:9.3.21.v20170918]
	at org.eclipse.jetty.servlets.CrossOriginFilter.handle(CrossOriginFilter.java:308) [162:org.eclipse.jetty.servlets:9.3.21.v20170918]
	at org.eclipse.jetty.servlets.CrossOriginFilter.doFilter(CrossOriginFilter.java:262) [162:org.eclipse.jetty.servlets:9.3.21.v20170918]
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759) [161:org.eclipse.jetty.servlet:9.3.21.v20170918]
	at org.eclipse.jetty.websocket.server.WebSocketUpgradeFilter.doFilter(WebSocketUpgradeFilter.java:205) [171:org.eclipse.jetty.websocket.server:9.3.21.v20170918]
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759) [161:org.eclipse.jetty.servlet:9.3.21.v20170918]
	at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:61) [142:org.apache.shiro.web:1.3.2]
	at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108) [142:org.apache.shiro.web:1.3.2]
	at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137) [142:org.apache.shiro.web:1.3.2]
	at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125) [142:org.apache.shiro.web:1.3.2]
	at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66) [142:org.apache.shiro.web:1.3.2]
	at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108) [142:org.apache.shiro.web:1.3.2]
	at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137) [142:org.apache.shiro.web:1.3.2]
	at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125) [142:org.apache.shiro.web:1.3.2]
	at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66) [142:org.apache.shiro.web:1.3.2]
	at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449) [142:org.apache.shiro.web:1.3.2]
	at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365) [142:org.apache.shiro.web:1.3.2]
	at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90) [141:org.apache.shiro.core:1.3.2]
	at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83) [141:org.apache.shiro.core:1.3.2]
	at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383) [141:org.apache.shiro.core:1.3.2]
	at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362) [142:org.apache.shiro.web:1.3.2]
	at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125) [142:org.apache.shiro.web:1.3.2]
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1751) [161:org.eclipse.jetty.servlet:9.3.21.v20170918]
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582) [161:org.eclipse.jetty.servlet:9.3.21.v20170918]
	at org.ops4j.pax.web.service.jetty.internal.HttpServiceServletHandler.doHandle(HttpServiceServletHandler.java:71) [463:org.ops4j.pax.web.pax-web-jetty:6.0.9]
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) [160:org.eclipse.jetty.server:9.3.21.v20170918]
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) [158:org.eclipse.jetty.security:9.3.21.v20170918]
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) [160:org.eclipse.jetty.server:9.3.21.v20170918]
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) [160:org.eclipse.jetty.server:9.3.21.v20170918]
	at org.ops4j.pax.web.service.jetty.internal.HttpServiceContext.doHandle(HttpServiceContext.java:284) [463:org.ops4j.pax.web.pax-web-jetty:6.0.9]
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512) [161:org.eclipse.jetty.servlet:9.3.21.v20170918]
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) [160:org.eclipse.jetty.server:9.3.21.v20170918]
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) [160:org.eclipse.jetty.server:9.3.21.v20170918]
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) [160:org.eclipse.jetty.server:9.3.21.v20170918]
	at org.ops4j.pax.web.service.jetty.internal.JettyServerHandlerCollection.handle(JettyServerHandlerCollection.java:80) [463:org.ops4j.pax.web.pax-web-jetty:6.0.9]
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) [160:org.eclipse.jetty.server:9.3.21.v20170918]
	at org.eclipse.jetty.server.Server.handle(Server.java:534) [160:org.eclipse.jetty.server:9.3.21.v20170918]
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333) [160:org.eclipse.jetty.server:9.3.21.v20170918]
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) [160:org.eclipse.jetty.server:9.3.21.v20170918]
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283) [152:org.eclipse.jetty.io:9.3.21.v20170918]
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108) [152:org.eclipse.jetty.io:9.3.21.v20170918]
	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) [152:org.eclipse.jetty.io:9.3.21.v20170918]
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) [163:org.eclipse.jetty.util:9.3.21.v20170918]
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) [163:org.eclipse.jetty.util:9.3.21.v20170918]
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) [163:org.eclipse.jetty.util:9.3.21.v20170918]
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) [163:org.eclipse.jetty.util:9.3.21.v20170918]
	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) [163:org.eclipse.jetty.util:9.3.21.v20170918]
	at java.lang.Thread.run(Thread.java:748) [?:?]
Caused by: org.opendaylight.controller.md.sal.common.api.data.TransactionCommitFailedException: commit execution failed
	at org.opendaylight.controller.md.sal.dom.broker.impl.TransactionCommitFailedExceptionMapper.newWithCause(TransactionCommitFailedExceptionMapper.java:37) ~[?:?]
	at org.opendaylight.controller.md.sal.dom.broker.impl.TransactionCommitFailedExceptionMapper.newWithCause(TransactionCommitFailedExceptionMapper.java:19) ~[?:?]
	at org.opendaylight.yangtools.util.concurrent.ExceptionMapper.apply(ExceptionMapper.java:91) ~[?:?]
	at org.opendaylight.controller.md.sal.dom.broker.impl.TransactionCommitFailedExceptionMapper.apply(TransactionCommitFailedExceptionMapper.java:43) ~[?:?]
	at org.opendaylight.controller.md.sal.dom.broker.impl.TransactionCommitFailedExceptionMapper.apply(TransactionCommitFailedExceptionMapper.java:19) ~[?:?]
	at org.opendaylight.mdsal.common.api.MappingCheckedFuture.mapException(MappingCheckedFuture.java:62) ~[?:?]
	... 77 more
Caused by: org.opendaylight.mdsal.common.api.OptimisticLockFailedException: Optimistic lock failed.
	at org.opendaylight.controller.cluster.datastore.ShardDataTree.lambda$processNextPendingTransaction$0(ShardDataTree.java:740) ~[?:?]
	at org.opendaylight.controller.cluster.datastore.ShardDataTree.processNextPending(ShardDataTree.java:778) ~[?:?]
	at org.opendaylight.controller.cluster.datastore.ShardDataTree.processNextPendingTransaction(ShardDataTree.java:725) ~[?:?]
	at org.opendaylight.controller.cluster.datastore.ShardDataTree.startCanCommit(ShardDataTree.java:808) ~[?:?]
	at org.opendaylight.controller.cluster.datastore.SimpleShardDataTreeCohort.canCommit(SimpleShardDataTreeCohort.java:84) ~[?:?]
	at org.opendaylight.controller.cluster.datastore.CohortEntry.canCommit(CohortEntry.java:97) ~[?:?]
	at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator.handleCanCommit(ShardCommitCoordinator.java:236) ~[?:?]
	at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator.handleReadyLocalTransaction(ShardCommitCoordinator.java:200) ~[?:?]
	at org.opendaylight.controller.cluster.datastore.Shard.handleReadyLocalTransaction(Shard.java:731) ~[?:?]
	at org.opendaylight.controller.cluster.datastore.Shard.handleNonRaftCommand(Shard.java:333) ~[?:?]
	at org.opendaylight.controller.cluster.raft.RaftActor.handleCommand(RaftActor.java:270) ~[?:?]
	at org.opendaylight.controller.cluster.common.actor.AbstractUntypedPersistentActor.onReceiveCommand(AbstractUntypedPersistentActor.java:44) ~[?:?]
	at akka.persistence.UntypedPersistentActor.onReceive(PersistentActor.scala:275) ~[?:?]
	at org.opendaylight.controller.cluster.common.actor.MeteringBehavior.apply(MeteringBehavior.java:104) ~[?:?]
	at akka.actor.ActorCell$$anonfun$become$1.applyOrElse(ActorCell.scala:608) ~[?:?]
	at akka.actor.Actor.aroundReceive(Actor.scala:517) ~[?:?]
	at akka.actor.Actor.aroundReceive$(Actor.scala:515) ~[?:?]
	at akka.persistence.UntypedPersistentActor.akka$persistence$Eventsourced$$super$aroundReceive(PersistentActor.scala:273) ~[?:?]
	at akka.persistence.Eventsourced$$anon$1.stateReceive(Eventsourced.scala:691) ~[?:?]
	at akka.persistence.Eventsourced.aroundReceive(Eventsourced.scala:192) ~[?:?]
	at akka.persistence.Eventsourced.aroundReceive$(Eventsourced.scala:191) ~[?:?]
	at akka.persistence.UntypedPersistentActor.aroundReceive(PersistentActor.scala:273) ~[?:?]
	at akka.actor.ActorCell.receiveMessage(ActorCell.scala:590) ~[?:?]
	at akka.actor.ActorCell.invoke(ActorCell.scala:559) ~[?:?]
	at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257) ~[?:?]
	at akka.dispatch.Mailbox.run(Mailbox.scala:224) ~[?:?]
	at akka.dispatch.Mailbox.exec(Mailbox.scala:234) ~[?:?]
	at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) ~[?:?]
	at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) ~[?:?]
	at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) ~[?:?]
	at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) ~[?:?]
Caused by: org.opendaylight.yangtools.yang.data.api.schema.tree.ConflictingModificationAppliedException: Node was deleted by other transaction.
	at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkWriteApplicable(SchemaAwareApplyOperation.java:180) ~[?:?]
	at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:135) ~[?:?]
	at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkChildPreconditions(AbstractNodeContainerModificationStrategy.java:307) ~[?:?]
	at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkMergeApplicable(AbstractNodeContainerModificationStrategy.java:315) ~[?:?]
	at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:138) ~[?:?]
	at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkChildPreconditions(AbstractNodeContainerModificationStrategy.java:307) ~[?:?]
	at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkMergeApplicable(AbstractNodeContainerModificationStrategy.java:315) ~[?:?]
	at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:138) ~[?:?]
	at org.opendaylight.yangtools.yang.data.impl.schema.tree.StructuralContainerModificationStrategy.checkApplicable(StructuralContainerModificationStrategy.java:101) ~[?:?]
	at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkChildPreconditions(AbstractNodeContainerModificationStrategy.java:307) ~[?:?]
	at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkMergeApplicable(AbstractNodeContainerModificationStrategy.java:315) ~[?:?]
	at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:138) ~[?:?]
	at org.opendaylight.yangtools.yang.data.impl.schema.tree.StructuralContainerModificationStrategy.checkApplicable(StructuralContainerModificationStrategy.java:101) ~[?:?]
	at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkChildPreconditions(AbstractNodeContainerModificationStrategy.java:307) ~[?:?]
	at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkTouchApplicable(AbstractNodeContainerModificationStrategy.java:290) ~[?:?]
	at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:132) ~[?:?]
	at org.opendaylight.yangtools.yang.data.impl.schema.tree.RootModificationApplyOperation.checkApplicable(RootModificationApplyOperation.java:72) ~[?:?]
	at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractDataTreeTip.validate(AbstractDataTreeTip.java:35) ~[?:?]
	at org.opendaylight.controller.cluster.datastore.ShardDataTree.lambda$processNextPendingTransaction$0(ShardDataTree.java:732) ~[?:?]
	at org.opendaylight.controller.cluster.datastore.ShardDataTree.processNextPending(ShardDataTree.java:778) ~[?:?]
	at org.opendaylight.controller.cluster.datastore.ShardDataTree.processNextPendingTransaction(ShardDataTree.java:725) ~[?:?]
	at org.opendaylight.controller.cluster.datastore.ShardDataTree.startCanCommit(ShardDataTree.java:808) ~[?:?]
	at org.opendaylight.controller.cluster.datastore.SimpleShardDataTreeCohort.canCommit(SimpleShardDataTreeCohort.java:84) ~[?:?]
	at org.opendaylight.controller.cluster.datastore.CohortEntry.canCommit(CohortEntry.java:97) ~[?:?]
	at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator.handleCanCommit(ShardCommitCoordinator.java:236) ~[?:?]
	at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator.handleReadyLocalTransaction(ShardCommitCoordinator.java:200) ~[?:?]
	at org.opendaylight.controller.cluster.datastore.Shard.handleReadyLocalTransaction(Shard.java:731) ~[?:?]
	at org.opendaylight.controller.cluster.datastore.Shard.handleNonRaftCommand(Shard.java:333) ~[?:?]
	at org.opendaylight.controller.cluster.raft.RaftActor.handleCommand(RaftActor.java:270) ~[?:?]
	at org.opendaylight.controller.cluster.common.actor.AbstractUntypedPersistentActor.onReceiveCommand(AbstractUntypedPersistentActor.java:44) ~[?:?]
	at akka.persistence.UntypedPersistentActor.onReceive(PersistentActor.scala:275) ~[?:?]
	at org.opendaylight.controller.cluster.common.actor.MeteringBehavior.apply(MeteringBehavior.java:104) ~[?:?]
	at akka.actor.ActorCell$$anonfun$become$1.applyOrElse(ActorCell.scala:608) ~[?:?]
	at akka.actor.Actor.aroundReceive(Actor.scala:517) ~[?:?]
	at akka.actor.Actor.aroundReceive$(Actor.scala:515) ~[?:?]
	at akka.persistence.UntypedPersistentActor.akka$persistence$Eventsourced$$super$aroundReceive(PersistentActor.scala:273) ~[?:?]
	at akka.persistence.Eventsourced$$anon$1.stateReceive(Eventsourced.scala:691) ~[?:?]
	at akka.persistence.Eventsourced.aroundReceive(Eventsourced.scala:192) ~[?:?]
	at akka.persistence.Eventsourced.aroundReceive$(Eventsourced.scala:191) ~[?:?]
	at akka.persistence.UntypedPersistentActor.aroundReceive(PersistentActor.scala:273) ~[?:?]
	at akka.actor.ActorCell.receiveMessage(ActorCell.scala:590) ~[?:?]
	at akka.actor.ActorCell.invoke(ActorCell.scala:559) ~[?:?]
	at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257) ~[?:?]
	at akka.dispatch.Mailbox.run(Mailbox.scala:224) ~[?:?]
	at akka.dispatch.Mailbox.exec(Mailbox.scala:234) ~[?:?]
	at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) ~[?:?]
	at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) ~[?:?]
	at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) ~[?:?]
	at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) ~[?:?]
Comment by Sam Hague [ 24/May/18 ]

still seen: https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-openstack-queens-upstream-stateful-oxygen/682/odl_1/odl1_karaf.log.gz

Comment by Michael Vorburger [ 25/Jun/18 ]

> not an ODL bug I mean that ODL behaved in the correct manner.
> It would be better if the exception did not appear in the log, of course.

I'm actively working on fixing this in ongoing https://git.opendaylight.org/gerrit/#/c/72735/ WIP...

jhershbe I don't want to "grab" this from you, but I'm assuming you're cool with it if I re-assign this one to me?

Comment by Sam Hague [ 25/Jun/18 ]

question on the comment about not showing the exception - the bug is that networking-odl is sending events in the wrong order. ODL can't do anything with this since the required resources aren't there when the request is made. BUt, if we don't log this exception, how will we know if the root problem is fixed?

Comment by Michael Vorburger [ 26/Jun/18 ]

> the bug is that networking-odl is sending events in the wrong order. ODL can't do anything with this since the required resources aren't there when the request is made.

well now that is of course the bigger more general problem we're after solving under NEUTRON-158... but aren't you mixing up 2 not directly related problems here? This bug is not (directly) about that, from what is written above, I thought. But if you think it is, then just make this Jira have a requires/depends on link to NEUTRON-158 - fine for me.

> BUt, if we don't log this exception, how will we know if the root problem is fixed?

c/72735 makes Neutron actually return a HTTP 500 error in case of an OptimisticLockFailedException to the driver (instead of logging an error on ODL side but return 201 to the driver, which is the current behaviour, which IMHO is completely wrong), so the driver will log it as a failed operation - and, I guess (not sure), eventually retry.

Comment by Michael Vorburger [ 02/Jul/18 ]

shague and jhershbe I was torn whether to really just send it to the Neutron drive to handle or log anyway, and more out of extra caution than because I think it's the right thing to do (actually I really think it's not...) at least WARN log it anyway in Neutron; so I'vehave proposed https://git.opendaylight.org/gerrit/#/c/73665 (adds logging), with https://git.opendaylight.org/gerrit/#/c/73666/ (supress warn log IFF OptimisticLockFailedException) as a possibly extension, for your review and discussion of that we really want to do here.

Generated at Wed Feb 07 20:25:42 UTC 2024 using Jira 8.20.10#820010-sha1:ace47f9899e9ee25d7157d59aa17ab06aee30d3d.