[YANGTOOLS-383] Clustering: ModifiedNodeDoesNotExistException on 1800 or more updates

Duplicate — Sun, 10 Apr 2022 18:35:26 +0000

When 1800 or more updates are sent to BGP with clustering enabled (3-cluster node, no persistence nor replication), at some point the ModifiedNodeDoesNotExistException will be logged:

org.opendaylight.yangtools.yang.data.api.schema.tree.ModifiedNodeDoesNotExistException: Node /(urn:opendaylight:params:xml:ns:yang:bgp-rib?revision=2013-09-25)bgp-rib/rib/rib[

{(urn:opendaylight:params:xml:ns:yang:bgp-rib?revision=2013-09-25)id=example-bgp-rib-1}

]/loc-rib/tables/tables[

{(urn:opendaylight:params:xml:ns:yang:bgp-rib?revision=2013-09-25)afi=(urn:opendaylight:params:xml:ns:yang:bgp-types?revision=2013-09-19)ipv4-address-family, (urn:opendaylight:params:xml:ns:yang:bgp-rib?revision=2013-09-25)safi=(urn:opendaylight:params:xml:ns:yang:bgp-types?revision=2013-09-19)unicast-subsequent-address-family}

]/routes/ipv4-routes/ipv4-route/ipv4-route[

{(urn:opendaylight:params:xml:ns:yang:bgp-rib?revision=2013-09-25)prefix=127.0.49.112/28}

] does not exist. Cannot apply modification to its children.
at org.opendaylight.yangtools.yang.data.impl.schema.tree.NormalizedNodeContainerModificationStrategy.checkSubtreeModificationApplicable(NormalizedNodeContainerModificationStrategy.java:161)[157:org.opendaylight.yangtools.yang-data-impl:0.7.0.SNAPSHOT]
at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:135)[157:org.opendaylight.yangtools.yang-data-impl:0.7.0.SNAPSHOT]
at org.opendaylight.yangtools.yang.data.impl.schema.tree.NormalizedNodeContainerModificationStrategy.checkChildPreconditions(NormalizedNodeContainerModificationStrategy.java:175)[157:org.opendaylight.yangtools.yang-data-impl:0.7.0.SNAPSHOT]
at org.opendaylight.yangtools.yang.data.impl.schema.tree.NormalizedNodeContainerModificationStrategy.checkMergeApplicable(NormalizedNodeContainerModificationStrategy.java:183)[157:org.opendaylight.yangtools.yang-data-impl:0.7.0.SNAPSHOT]
at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:141)[157:org.opendaylight.yangtools.yang-data-impl:0.7.0.SNAPSHOT]
at org.opendaylight.yangtools.yang.data.impl.schema.tree.NormalizedNodeContainerModificationStrategy.checkChildPreconditions(NormalizedNodeContainerModificationStrategy.java:175)[157:org.opendaylight.yangtools.yang-data-impl:0.7.0.SNAPSHOT]
at org.opendaylight.yangtools.yang.data.impl.schema.tree.NormalizedNodeContainerModificationStrategy.checkMergeApplicable(NormalizedNodeContainerModificationStrategy.java:183)[157:org.opendaylight.yangtools.yang-data-impl:0.7.0.SNAPSHOT]
at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:141)[157:org.opendaylight.yangtools.yang-data-impl:0.7.0.SNAPSHOT]
at org.opendaylight.yangtools.yang.data.impl.schema.tree.NormalizedNodeContainerModificationStrategy.checkChildPreconditions(NormalizedNodeContainerModificationStrategy.java:175)[157:org.opendaylight.yangtools.yang-data-impl:0.7.0.SNAPSHOT]
at org.opendaylight.yangtools.yang.data.impl.schema.tree.NormalizedNodeContainerModificationStrategy.checkMergeApplicable(NormalizedNodeContainerModificationStrategy.java:183)[157:org.opendaylight.yangtools.yang-data-impl:0.7.0.SNAPSHOT]
at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:141)[157:org.opendaylight.yangtools.yang-data-impl:0.7.0.SNAPSHOT]
at org.opendaylight.yangtools.yang.data.impl.schema.tree.NormalizedNodeContainerModificationStrategy.checkChildPreconditions(NormalizedNodeContainerModificationStrategy.java:175)[157:org.opendaylight.yangtools.yang-data-impl:0.7.0.SNAPSHOT]
at org.opendaylight.yangtools.yang.data.impl.schema.tree.NormalizedNodeContainerModificationStrategy.checkMergeApplicable(NormalizedNodeContainerModificationStrategy.java:183)[157:org.opendaylight.yangtools.yang-data-impl:0.7.0.SNAPSHOT]
at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:141)[157:org.opendaylight.yangtools.yang-data-impl:0.7.0.SNAPSHOT]
at org.opendaylight.yangtools.yang.data.impl.schema.tree.NormalizedNodeContainerModificationStrategy.checkChildPreconditions(NormalizedNodeContainerModificationStrategy.java:175)[157:org.opendaylight.yangtools.yang-data-impl:0.7.0.SNAPSHOT]
at org.opendaylight.yangtools.yang.data.impl.schema.tree.NormalizedNodeContainerModificationStrategy.checkMergeApplicable(NormalizedNodeContainerModificationStrategy.java:183)[157:org.opendaylight.yangtools.yang-data-impl:0.7.0.SNAPSHOT]
at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:141)[157:org.opendaylight.yangtools.yang-data-impl:0.7.0.SNAPSHOT]
at org.opendaylight.yangtools.yang.data.impl.schema.tree.NormalizedNodeContainerModificationStrategy.checkChildPreconditions(NormalizedNodeContainerModificationStrategy.java:175)[157:org.opendaylight.yangtools.yang-data-impl:0.7.0.SNAPSHOT]
at org.opendaylight.yangtools.yang.data.impl.schema.tree.NormalizedNodeContainerModificationStrategy.checkMergeApplicable(NormalizedNodeContainerModificationStrategy.java:183)[157:org.opendaylight.yangtools.yang-data-impl:0.7.0.SNAPSHOT]
at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:141)[157:org.opendaylight.yangtools.yang-data-impl:0.7.0.SNAPSHOT]
at org.opendaylight.yangtools.yang.data.impl.schema.tree.NormalizedNodeContainerModificationStrategy.checkChildPreconditions(NormalizedNodeContainerModificationStrategy.java:175)[157:org.opendaylight.yangtools.yang-data-impl:0.7.0.SNAPSHOT]
at org.opendaylight.yangtools.yang.data.impl.schema.tree.NormalizedNodeContainerModificationStrategy.checkMergeApplicable(NormalizedNodeContainerModificationStrategy.java:183)[157:org.opendaylight.yangtools.yang-data-impl:0.7.0.SNAPSHOT]
at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:141)[157:org.opendaylight.yangtools.yang-data-impl:0.7.0.SNAPSHOT]
at org.opendaylight.yangtools.yang.data.impl.schema.tree.NormalizedNodeContainerModificationStrategy.checkChildPreconditions(NormalizedNodeContainerModificationStrategy.java:175)[157:org.opendaylight.yangtools.yang-data-impl:0.7.0.SNAPSHOT]
at org.opendaylight.yangtools.yang.data.impl.schema.tree.NormalizedNodeContainerModificationStrategy.checkMergeApplicable(NormalizedNodeContainerModificationStrategy.java:183)[157:org.opendaylight.yangtools.yang-data-impl:0.7.0.SNAPSHOT]
at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:141)[157:org.opendaylight.yangtools.yang-data-impl:0.7.0.SNAPSHOT]
at org.opendaylight.yangtools.yang.data.impl.schema.tree.NormalizedNodeContainerModificationStrategy.checkChildPreconditions(NormalizedNodeContainerModificationStrategy.java:175)[157:org.opendaylight.yangtools.yang-data-impl:0.7.0.SNAPSHOT]
at org.opendaylight.yangtools.yang.data.impl.schema.tree.NormalizedNodeContainerModificationStrategy.checkMergeApplicable(NormalizedNodeContainerModificationStrategy.java:183)[157:org.opendaylight.yangtools.yang-data-impl:0.7.0.SNAPSHOT]
at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:141)[157:org.opendaylight.yangtools.yang-data-impl:0.7.0.SNAPSHOT]
at org.opendaylight.yangtools.yang.data.impl.schema.tree.NormalizedNodeContainerModificationStrategy.checkChildPreconditions(NormalizedNodeContainerModificationStrategy.java:175)[157:org.opendaylight.yangtools.yang-data-impl:0.7.0.SNAPSHOT]
at org.opendaylight.yangtools.yang.data.impl.schema.tree.NormalizedNodeContainerModificationStrategy.checkSubtreeModificationApplicable(NormalizedNodeContainerModificationStrategy.java:165)[157:org.opendaylight.yangtools.yang-data-impl:0.7.0.SNAPSHOT]
at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:135)[157:org.opendaylight.yangtools.yang-data-impl:0.7.0.SNAPSHOT]
at org.opendaylight.yangtools.yang.data.impl.schema.tree.RootModificationApplyOperation.checkApplicable(RootModificationApplyOperation.java:72)[157:org.opendaylight.yangtools.yang-data-impl:0.7.0.SNAPSHOT]
at org.opendaylight.yangtools.yang.data.impl.schema.tree.InMemoryDataTree.validate(InMemoryDataTree.java:77)[157:org.opendaylight.yangtools.yang-data-impl:0.7.0.SNAPSHOT]
at org.opendaylight.controller.md.sal.dom.store.impl.InMemoryDOMDataStore$ThreePhaseCommitImpl.canCommit(InMemoryDOMDataStore.java:232)[227:org.opendaylight.controller.sal-inmemory-datastore:1.2.0.SNAPSHOT]
at org.opendaylight.controller.md.sal.dom.store.impl.ChainedTransactionCommitImpl.canCommit(ChainedTransactionCommitImpl.java:30)[227:org.opendaylight.controller.sal-inmemory-datastore:1.2.0.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator.doCanCommit(ShardCommitCoordinator.java:138)[249:org.opendaylight.controller.sal-distributed-datastore:1.2.0.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator.handleCanCommit(ShardCommitCoordinator.java:128)[249:org.opendaylight.controller.sal-distributed-datastore:1.2.0.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.Shard.handleCanCommitTransaction(Shard.java:389)[249:org.opendaylight.controller.sal-distributed-datastore:1.2.0.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.Shard.onReceiveCommand(Shard.java:252)[249:org.opendaylight.controller.sal-distributed-datastore:1.2.0.SNAPSHOT]
at akka.persistence.UntypedPersistentActor.onReceive(Eventsourced.scala:430)[239:com.typesafe.akka.persistence.experimental:2.3.4]
at org.opendaylight.controller.cluster.common.actor.MeteringBehavior.apply(MeteringBehavior.java:96)[241:org.opendaylight.controller.sal-clustering-commons:1.2.0.SNAPSHOT]
at akka.actor.ActorCell$$anonfun$become$1.applyOrElse(ActorCell.scala:534)[234:com.typesafe.akka.actor:2.3.4]
at akka.persistence.Recovery$State$class.process(Recovery.scala:30)[239:com.typesafe.akka.persistence.experimental:2.3.4]
at akka.persistence.ProcessorImpl$$anon$2.process(Processor.scala:103)[239:com.typesafe.akka.persistence.experimental:2.3.4]
at akka.persistence.ProcessorImpl$$anon$2.aroundReceive(Processor.scala:114)[239:com.typesafe.akka.persistence.experimental:2.3.4]
at akka.persistence.Recovery$class.aroundReceive(Recovery.scala:256)[239:com.typesafe.akka.persistence.experimental:2.3.4]
at akka.persistence.UntypedPersistentActor.akka$persistence$Eventsourced$$super$aroundReceive(Eventsourced.scala:428)[239:com.typesafe.akka.persistence.experimental:2.3.4]
at akka.persistence.Eventsourced$$anon$2.doAroundReceive(Eventsourced.scala:82)[239:com.typesafe.akka.persistence.experimental:2.3.4]
at akka.persistence.Eventsourced$$anon$2.aroundReceive(Eventsourced.scala:78)[239:com.typesafe.akka.persistence.experimental:2.3.4]
at akka.persistence.Eventsourced$class.aroundReceive(Eventsourced.scala:369)[239:com.typesafe.akka.persistence.experimental:2.3.4]
at akka.persistence.UntypedPersistentActor.aroundReceive(Eventsourced.scala:428)[239:com.typesafe.akka.persistence.experimental:2.3.4]
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)[234:com.typesafe.akka.actor:2.3.4]
at akka.actor.ActorCell.invoke(ActorCell.scala:487)[234:com.typesafe.akka.actor:2.3.4]
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)[234:com.typesafe.akka.actor:2.3.4]
at akka.dispatch.Mailbox.run(Mailbox.scala:220)[234:com.typesafe.akka.actor:2.3.4]
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393)[234:com.typesafe.akka.actor:2.3.4]
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)[231:org.scala-lang.scala-library:2.10.4.v20140209-180020-VFINAL-b66a39653b]
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)[231:org.scala-lang.scala-library:2.10.4.v20140209-180020-VFINAL-b66a39653b]
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)[231:org.scala-lang.scala-library:2.10.4.v20140209-180020-VFINAL-b66a39653b]
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)[231:org.scala-lang.scala-library:2.10.4.v20140209-180020-VFINAL-b66a39653b]

Once that happens, all further attempts to do any transactions will fail with the following exception:

java.util.concurrent.ExecutionException: TransactionCommitFailedException{message=Data did not pass validation., errorList=[RpcError [message=Data did not pass validation., severity=ERROR, errorType=APPLICATION, tag=operation-failed, applicationTag=null, info=null, cause=org.opendaylight.yangtools.yang.data.api.schema.tree.ModifiedNodeDoesNotExistException: Node /(urn:opendaylight:params:xml:ns:yang:bgp-rib?revision=2013-09-25)bgp-rib/rib/rib[

{(urn:opendaylight:params:xml:ns:yang:bgp-rib?revision=2013-09-25)id=example-bgp-rib-1}

]/loc-rib/tables/tables[

]/routes/ipv4-routes/ipv4-route/ipv4-route[

{(urn:opendaylight:params:xml:ns:yang:bgp-rib?revision=2013-09-25)prefix=127.0.49.112/28}

] does not exist. Cannot apply modification to its children.]]}
at com.google.common.util.concurrent.Futures$ImmediateFailedFuture.get(Futures.java:183)[134:com.google.guava:14.0.1]
at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator.doCanCommit(ShardCommitCoordinator.java:138)[249:org.opendaylight.controller.sal-distributed-datastore:1.2.0.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator.handleCanCommit(ShardCommitCoordinator.java:128)[249:org.opendaylight.controller.sal-distributed-datastore:1.2.0.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.Shard.handleCanCommitTransaction(Shard.java:389)[249:org.opendaylight.controller.sal-distributed-datastore:1.2.0.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.Shard.onReceiveCommand(Shard.java:252)[249:org.opendaylight.controller.sal-distributed-datastore:1.2.0.SNAPSHOT]
at akka.persistence.UntypedPersistentActor.onReceive(Eventsourced.scala:430)[239:com.typesafe.akka.persistence.experimental:2.3.4]
at org.opendaylight.controller.cluster.common.actor.MeteringBehavior.apply(MeteringBehavior.java:96)[241:org.opendaylight.controller.sal-clustering-commons:1.2.0.SNAPSHOT]
at akka.actor.ActorCell$$anonfun$become$1.applyOrElse(ActorCell.scala:534)[234:com.typesafe.akka.actor:2.3.4]
at akka.persistence.Recovery$State$class.process(Recovery.scala:30)[239:com.typesafe.akka.persistence.experimental:2.3.4]
at akka.persistence.ProcessorImpl$$anon$2.process(Processor.scala:103)[239:com.typesafe.akka.persistence.experimental:2.3.4]
at akka.persistence.ProcessorImpl$$anon$2.aroundReceive(Processor.scala:114)[239:com.typesafe.akka.persistence.experimental:2.3.4]
at akka.persistence.Recovery$class.aroundReceive(Recovery.scala:256)[239:com.typesafe.akka.persistence.experimental:2.3.4]
at akka.persistence.UntypedPersistentActor.akka$persistence$Eventsourced$$super$aroundReceive(Eventsourced.scala:428)[239:com.typesafe.akka.persistence.experimental:2.3.4]
at akka.persistence.Eventsourced$$anon$2.doAroundReceive(Eventsourced.scala:82)[239:com.typesafe.akka.persistence.experimental:2.3.4]
at akka.persistence.Eventsourced$$anon$2.aroundReceive(Eventsourced.scala:78)[239:com.typesafe.akka.persistence.experimental:2.3.4]
at akka.persistence.Eventsourced$class.aroundReceive(Eventsourced.scala:369)[239:com.typesafe.akka.persistence.experimental:2.3.4]
at akka.persistence.UntypedPersistentActor.aroundReceive(Eventsourced.scala:428)[239:com.typesafe.akka.persistence.experimental:2.3.4]
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)[234:com.typesafe.akka.actor:2.3.4]
at akka.actor.ActorCell.invoke(ActorCell.scala:487)[234:com.typesafe.akka.actor:2.3.4]
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)[234:com.typesafe.akka.actor:2.3.4]
at akka.dispatch.Mailbox.run(Mailbox.scala:220)[234:com.typesafe.akka.actor:2.3.4]
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393)[234:com.typesafe.akka.actor:2.3.4]
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)[231:org.scala-lang.scala-library:2.10.4.v20140209-180020-VFINAL-b66a39653b]
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)[231:org.scala-lang.scala-library:2.10.4.v20140209-180020-VFINAL-b66a39653b]
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)[231:org.scala-lang.scala-library:2.10.4.v20140209-180020-VFINAL-b66a39653b]
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)[231:org.scala-lang.scala-library:2.10.4.v20140209-180020-VFINAL-b66a39653b]
Caused by: TransactionCommitFailedException{message=Data did not pass validation., errorList=[RpcError [message=Data did not pass validation., severity=ERROR, errorType=APPLICATION, tag=operation-failed, applicationTag=null, info=null, cause=org.opendaylight.yangtools.yang.data.api.schema.tree.ModifiedNodeDoesNotExistException: Node /(urn:opendaylight:params:xml:ns:yang:bgp-rib?revision=2013-09-25)bgp-rib/rib/rib[

{(urn:opendaylight:params:xml:ns:yang:bgp-rib?revision=2013-09-25)id=example-bgp-rib-1}

]/loc-rib/tables/tables[

]/routes/ipv4-routes/ipv4-route/ipv4-route[

{(urn:opendaylight:params:xml:ns:yang:bgp-rib?revision=2013-09-25)prefix=127.0.49.112/28}

] does not exist. Cannot apply modification to its children.]]}
at org.opendaylight.controller.md.sal.dom.store.impl.InMemoryDOMDataStore$ThreePhaseCommitImpl.canCommit(InMemoryDOMDataStore.java:244)[227:org.opendaylight.controller.sal-inmemory-datastore:1.2.0.SNAPSHOT]
at org.opendaylight.controller.md.sal.dom.store.impl.ChainedTransactionCommitImpl.canCommit(ChainedTransactionCommitImpl.java:30)[227:org.opendaylight.controller.sal-inmemory-datastore:1.2.0.SNAPSHOT]
... 25 more
Caused by: org.opendaylight.yangtools.yang.data.api.schema.tree.ModifiedNodeDoesNotExistException: Node /(urn:opendaylight:params:xml:ns:yang:bgp-rib?revision=2013-09-25)bgp-rib/rib/rib[

{(urn:opendaylight:params:xml:ns:yang:bgp-rib?revision=2013-09-25)id=example-bgp-rib-1}

]/loc-rib/tables/tables[

]/routes/ipv4-routes/ipv4-route/ipv4-route[

{(urn:opendaylight:params:xml:ns:yang:bgp-rib?revision=2013-09-25)prefix=127.0.49.112/28}

OpenDaylight JIRA

[YANGTOOLS-383] Clustering: ModifiedNodeDoesNotExistException on 1800 or more updates