[OPNFLWPLUG-1002] Transaction chain failed, recreating chain due toorg.opendaylight.controller.md.sal.common.api.data.TransactionCommitFailedException: Data did not pass validation Created: 19/Apr/18 Updated: 15/Oct/18 Resolved: 15/Oct/18 |
|
| Status: | Resolved |
| Project: | OpenFlowPlugin |
| Component/s: | statistics-manager |
| Affects Version/s: | Oxygen |
| Fix Version/s: | Fluorine-SR2, Neon |
| Type: | Bug | Priority: | Medium |
| Reporter: | Sai Sindhur Malleni | Assignee: | Anil Vishnoi |
| Resolution: | Cannot Reproduce | Votes: | 1 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Description |
|
In a clustered ODL setup with OpenStack, when trying to create VMs, attach FIPs and ping them, seeing several errors like this: 2018-04-19T17:25:16,882 | WARN | CommitFutures-4 | TransactionChainManager | 385 - org.opendaylight.openflowplugin.common - 0.6.0.redhat-5 | Transaction chain failed, recreating chain due to org.opendaylight.controller.md.sal.common.api.data.
TransactionCommitFailedException: Data did not pass validation.
at org.opendaylight.controller.cluster.datastore.ShardDataTree.lambda$processNextPendingTransaction$0(ShardDataTree.java:749) ~[233:org.opendaylight.controller.sal-distributed-datastore:1.7.0.redhat-5]
at org.opendaylight.controller.cluster.datastore.ShardDataTree.processNextPending(ShardDataTree.java:778) ~[233:org.opendaylight.controller.sal-distributed-datastore:1.7.0.redhat-5]
at org.opendaylight.controller.cluster.datastore.ShardDataTree.processNextPendingTransaction(ShardDataTree.java:725) ~[233:org.opendaylight.controller.sal-distributed-datastore:1.7.0.redhat-5]
at org.opendaylight.controller.cluster.datastore.ShardDataTree.startCanCommit(ShardDataTree.java:808) ~[233:org.opendaylight.controller.sal-distributed-datastore:1.7.0.redhat-5]
at org.opendaylight.controller.cluster.datastore.SimpleShardDataTreeCohort.canCommit(SimpleShardDataTreeCohort.java:84) ~[233:org.opendaylight.controller.sal-distributed-datastore:1.7.0.redhat-5]
at org.opendaylight.controller.cluster.datastore.ChainedCommitCohort.canCommit(ChainedCommitCohort.java:58) ~[233:org.opendaylight.controller.sal-distributed-datastore:1.7.0.redhat-5]
at org.opendaylight.controller.cluster.datastore.CohortEntry.canCommit(CohortEntry.java:97) ~[233:org.opendaylight.controller.sal-distributed-datastore:1.7.0.redhat-5]
at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator.handleCanCommit(ShardCommitCoordinator.java:236) ~[233:org.opendaylight.controller.sal-distributed-datastore:1.7.0.redhat-5]
at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator.handleForwardedReadyTransaction(ShardCommitCoordinator.java:114) ~[233:org.opendaylight.controller.sal-distributed-datastore:1.7.0.redhat-5]
at org.opendaylight.controller.cluster.datastore.Shard.handleForwardedReadyTransaction(Shard.java:755) ~[233:org.opendaylight.controller.sal-distributed-datastore:1.7.0.redhat-5]
at org.opendaylight.controller.cluster.datastore.Shard.handleNonRaftCommand(Shard.java:331) ~[233:org.opendaylight.controller.sal-distributed-datastore:1.7.0.redhat-5]
at org.opendaylight.controller.cluster.raft.RaftActor.handleCommand(RaftActor.java:270) ~[218:org.opendaylight.controller.sal-akka-raft:1.7.0.redhat-5]
at org.opendaylight.controller.cluster.common.actor.AbstractUntypedPersistentActor.onReceiveCommand(AbstractUntypedPersistentActor.java:44) ~[226:org.opendaylight.controller.sal-clustering-commons:1.7.0.redhat-5]
at akka.persistence.UntypedPersistentActor.onReceive(PersistentActor.scala:274) ~[43:com.typesafe.akka.persistence:2.5.4]
at org.opendaylight.controller.cluster.common.actor.MeteringBehavior.apply(MeteringBehavior.java:104) ~[226:org.opendaylight.controller.sal-clustering-commons:1.7.0.redhat-5]
at akka.actor.ActorCell$$anonfun$become$1.applyOrElse(ActorCell.scala:545) ~[40:com.typesafe.akka.actor:2.5.4]
at akka.actor.Actor.aroundReceive(Actor.scala:514) ~[40:com.typesafe.akka.actor:2.5.4]
at akka.actor.Actor.aroundReceive$(Actor.scala:512) ~[40:com.typesafe.akka.actor:2.5.4]
at akka.persistence.UntypedPersistentActor.akka$persistence$Eventsourced$$super$aroundReceive(PersistentActor.scala:272) ~[43:com.typesafe.akka.persistence:2.5.4]
at akka.persistence.Eventsourced$$anon$1.stateReceive(Eventsourced.scala:663) ~[43:com.typesafe.akka.persistence:2.5.4]
at akka.persistence.Eventsourced.aroundReceive(Eventsourced.scala:183) ~[43:com.typesafe.akka.persistence:2.5.4]
at akka.persistence.Eventsourced.aroundReceive$(Eventsourced.scala:182) ~[43:com.typesafe.akka.persistence:2.5.4]
at akka.persistence.UntypedPersistentActor.aroundReceive(PersistentActor.scala:272) ~[43:com.typesafe.akka.persistence:2.5.4]
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:527) ~[40:com.typesafe.akka.actor:2.5.4]
at akka.actor.ActorCell.invoke(ActorCell.scala:496) ~[40:com.typesafe.akka.actor:2.5.4]
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257) ~[40:com.typesafe.akka.actor:2.5.4]
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257) ~[40:com.typesafe.akka.actor:2.5.4]
at akka.dispatch.Mailbox.run(Mailbox.scala:224) ~[40:com.typesafe.akka.actor:2.5.4]
at akka.dispatch.Mailbox.exec(Mailbox.scala:234) ~[40:com.typesafe.akka.actor:2.5.4]
at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) ~[40:com.typesafe.akka.actor:2.5.4]
at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) ~[40:com.typesafe.akka.actor:2.5.4]
at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) ~[40:com.typesafe.akka.actor:2.5.4]
at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) ~[40:com.typesafe.akka.actor:2.5.4]
Caused by: org.opendaylight.yangtools.yang.data.api.schema.tree.ModifiedNodeDoesNotExistException: Node /(urn:opendaylight:inventory?revision=2013-08-19)nodes/node/node[\\{(urn:opendaylight:inventory?revision=2013-08-19)id=openflow:277753376228137}]/AugmentationIdentifier{childNames=[(urn:opendaylight:flow:inventory?revision=2013-08-19)description, (urn:opendaylight:flow:inventory?revision=2013-08-19)supported-actions, (urn:opendaylight:flow:inventory?revision=2013-08-19)hardware, (urn:opendaylight:flow:inventory?revision=2013-08-19)switch-features, (urn:opendaylight:flow:inventory?revision=2013-08-19)stale-meter, (urn:opendaylight:flow:inventory?revision=2013-08-19)supported-instructions, (urn:opendaylight:flow:inventory?revision=2013-08-19)meter, (urn:opendaylight:flow:inventory?revision=2013-08-19)serial-number, (urn:opendaylight:flow:inventory?revision=2013-08-19)stale-group, (urn:opendaylight:flow:inventory?revision=2013-08-19)supported-match-types, (urn:opendaylight:flow:inventory?revision=2013-08-19)port-number, (urn:opendaylight:flow:inventory?revision=2013-08-19)table, (urn:opendaylight:flow:inventory?revision=2013-08-19)group, (urn:opendaylight:flow:inventory?revision=2013-08-19)manufacturer, (urn:opendaylight:flow:inventory?revision=2013-08-19)table-features, (urn:opendaylight:flow:inventory?revision=2013-08-19)software, (urn:opendaylight:flow:inventory?revision=2013-08-19)ip-address]}/(urn:opendaylight:flow:inventory?revision=2013-08-19)group/group[\\{(urn:opendaylight:flow:inventory?revision=2013-08-19)group-id=210084}]
does not exist. Cannot apply modification to its children.
at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkTouchApplicable(AbstractNodeContainerModificationStrategy.java:283) ~[430:org.opendaylight.yangtools.yang-data-impl:2.0.1.redhat-5]
at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:132) ~[430:org.opendaylight.yangtools.yang-data-impl:2.0.1.redhat-5]
at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkChildPreconditions(AbstractNodeContainerModificationStrategy.java:307) ~[430:org.opendaylight.yangtools.yang-data-impl:2.0.1.redhat-5]
at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkMergeApplicable(AbstractNodeContainerModificationStrategy.java:315) ~[430:org.opendaylight.yangtools.yang-data-impl:2.0.1.redhat-5]
at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:138) ~[430:org.opendaylight.yangtools.yang-data-impl:2.0.1.redhat-5]
at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkChildPreconditions(AbstractNodeContainerModificationStrategy.java:307) ~[430:org.opendaylight.yangtools.yang-data-impl:2.0.1.redhat-5]
at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkTouchApplicable(AbstractNodeContainerModificationStrategy.java:290) ~[430:org.opendaylight.yangtools.yang-data-impl:2.0.1.redhat-5]
at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:132) ~[430:org.opendaylight.yangtools.yang-data-impl:2.0.1.redhat-5]
at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkChildPreconditions(AbstractNodeContainerModificationStrategy.java:307) ~[430:org.opendaylight.yangtools.yang-data-impl:2.0.1.redhat-5]
at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkTouchApplicable(AbstractNodeContainerModificationStrategy.java:290) ~[430:org.opendaylight.yangtools.yang-data-impl:2.0.1.redhat-5]
at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:132) ~[430:org.opendaylight.yangtools.yang-data-impl:2.0.1.redhat-5]
at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkChildPreconditions(AbstractNodeContainerModificationStrategy.java:307) ~[430:org.opendaylight.yangtools.yang-data-impl:2.0.1.redhat-5] redhat-5]
at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkChildPreconditions(AbstractNodeContainerModificationStrategy.java:307) ~[430:org.opendaylight.yangtools.yang-data-impl:2.0.1.redhat-5]
at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkTouchApplicable(AbstractNodeContainerModificationStrategy.java:290) ~[430:org.opendaylight.yangtools.yang-data-impl:2.0.1.redhat-5]
at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:132) ~[430:org.opendaylight.yangtools.yang-data-impl:2.0.1.redhat-5]
at org.opendaylight.yangtools.yang.data.impl.schema.tree.StructuralContainerModificationStrategy.checkApplicable(StructuralContainerModificationStrategy.java:101) ~[430:org.opendaylight.yangtools.yang-data-impl:2.0.1.redhat-5]
at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkChildPreconditions(AbstractNodeContainerModificationStrategy.java:307) ~[430:org.opendaylight.yangtools.yang-data-impl:2.0.1.redhat-5]
at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkTouchApplicable(AbstractNodeContainerModificationStrategy.java:290) ~[430:org.opendaylight.yangtools.yang-data-impl:2.0.1.redhat-5]
at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:132) ~[430:org.opendaylight.yangtools.yang-data-impl:2.0.1.redhat-5]
at org.opendaylight.yangtools.yang.data.impl.schema.tree.RootModificationApplyOperation.checkApplicable(RootModificationApplyOperation.java:72) ~[430:org.opendaylight.yangtools.yang-data-impl:2.0.1.redhat-5]
at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractDataTreeTip.validate(AbstractDataTreeTip.java:35) ~[430:org.opendaylight.yangtools.yang-data-impl:2.0.1.redhat-5]
at org.opendaylight.controller.cluster.datastore.ShardDataTree.lambda$processNextPendingTransaction$0(ShardDataTree.java:732) ~[233:org.opendaylight.controller.sal-distributed-datastore:1.7.0.redhat-5] ... 31 more 2018-04-19T17:25:16,888 | ERROR | CommitFutures-4 | TransactionChainManager | 385 - org.opendaylight.openflowplugin.common - 0.6.0.redhat-5 | Transaction commit failed. org.opendaylight.controller.md.sal.common.api.data.TransactionCommitFailedException: Data did not pass validation.
at org.opendaylight.controller.cluster.datastore.ShardDataTree.lambda$processNextPendingTransaction$0(ShardDataTree.java:749) ~[233:org.opendaylight.controller.sal-distributed-datastore:1.7.0.redhat-5]
at org.opendaylight.controller.cluster.datastore.ShardDataTree.processNextPending(ShardDataTree.java:778) ~[233:org.opendaylight.controller.sal-distributed-datastore:1.7.0.redhat-5]
at org.opendaylight.controller.cluster.datastore.ShardDataTree.processNextPendingTransaction(ShardDataTree.java:725) ~[233:org.opendaylight.controller.sal-distributed-datastore:1.7.0.redhat-5]
at org.opendaylight.controller.cluster.datastore.ShardDataTree.startCanCommit(ShardDataTree.java:808) ~[233:org.opendaylight.controller.sal-distributed-datastore:1.7.0.redhat-5]
at org.opendaylight.controller.cluster.datastore.SimpleShardDataTreeCohort.canCommit(SimpleShardDataTreeCohort.java:84) ~[233:org.opendaylight.controller.sal-distributed-datastore:1.7.0.redhat-5]
at org.opendaylight.controller.cluster.datastore.ChainedCommitCohort.canCommit(ChainedCommitCohort.java:58) ~[233:org.opendaylight.controller.sal-distributed-datastore:1.7.0.redhat-5]
at org.opendaylight.controller.cluster.datastore.CohortEntry.canCommit(CohortEntry.java:97) ~[233:org.opendaylight.controller.sal-distributed-datastore:1.7.0.redhat-5]
at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator.handleCanCommit(ShardCommitCoordinator.java:236) ~[233:org.opendaylight.controller.sal-distributed-datastore:1.7.0.redhat-5]
at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator.handleForwardedReadyTransaction(ShardCommitCoordinator.java:114) ~[233:org.opendaylight.controller.sal-distributed-datastore:1.7.0.redhat-5]
at org.opendaylight.controller.cluster.datastore.Shard.handleForwardedReadyTransaction(Shard.java:755) ~[233:org.opendaylight.controller.sal-distributed-datastore:1.7.0.redhat-5]
at org.opendaylight.controller.cluster.datastore.Shard.handleNonRaftCommand(Shard.java:331) ~[233:org.opendaylight.controller.sal-distributed-datastore:1.7.0.redhat-5]
at org.opendaylight.controller.cluster.raft.RaftActor.handleCommand(RaftActor.java:270) ~[218:org.opendaylight.controller.sal-akka-raft:1.7.0.redhat-5]
at org.opendaylight.controller.cluster.common.actor.AbstractUntypedPersistentActor.onReceiveCommand(AbstractUntypedPersistentActor.java:44) ~[226:org.opendaylight.controller.sal-clustering-commons:1.7.0.redhat-5]
|
| Comments |
| Comment by Sai Sindhur Malleni [ 19/Apr/18 ] |
|
I didn't seen any optimistic lock failed exceptions before this error. |
| Comment by Michael Vorburger [ 20/Apr/18 ] |
|
https://lists.opendaylight.org/pipermail/openflowplugin-dev/2018-April/008219.html |
| Comment by Anil Vishnoi [ 10/Sep/18 ] |
|
shague jluhrsen Do you still see this exception in your Oxygen/Fluorine CSIT ? |
| Comment by Jamo Luhrsen [ 10/Sep/18 ] |
|
smalleni, any chance you know if it was still showing up? Avishnoi, do we think something came in that would fix this? shague, do you know if we every saw this in our netvirt CSIT? if so, I can pull down logs from a bunch of jobs and search them. |
| Comment by Sam Hague [ 12/Sep/18 ] |
|
No, our exception checker has not been triggering on this exception. I see this jira component is statisticsmanager, but we disable that in the upstream csit. That might be why we don't see it. |
| Comment by Anil Vishnoi [ 15/Oct/18 ] |
|
jluhrsen Few issues are fixes in the relevant codebase, so it good to verify it.
org.opendaylight.openflowplugin.common - 0.6.0.redhat-5, it looks like you have seen this exception in your internal build ? I am closing this bug as of now, please re-open it if you see it again. |