https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/CONTROLLER-808 controller <p>These exceptions are seen on recovery which are preventing recovery from happening.</p> <p>2014-09-11 02:59:09,916 | WARN | lt-dispatcher-18 | ShardManager | 152 - com.typesafe.akka.slf4j - 2.3.4 | akka://opendaylight-cluster-data/user/shardmanager-config | Supervisor Strategy of resume applied<br/> at com.google.common.cache.LocalCache$Segment.getAndRecordStats(LocalCache.java:2412)<br/> at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2380)<br/> at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2342)<br/> at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2257)<br/> at com.google.common.cache.LocalCache.get(LocalCache.java:4000)<br/> at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:4004)<br/> at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4874)<br/> at org.opendaylight.yangtools.yang.data.impl.schema.tree.DataNodeContainerModificationStrategy.getChild(DataNodeContainerModificationStrategy.java:81)<br/> at org.opendaylight.yangtools.yang.data.impl.schema.tree.DataNodeContainerModificationStrategy$ContainerModificationStrategy.getChild(DataNodeContainerModificationStrategy.java:119)<br/> at org.opendaylight.yangtools.yang.data.impl.schema.tree.RootModificationApplyOperation.getChild(RootModificationApplyOperation.java:66)<br/> at org.opendaylight.yangtools.yang.data.impl.schema.tree.TreeNodeUtils.findNodeChecked(TreeNodeUtils.java:53)<br/> at org.opendaylight.yangtools.yang.data.impl.schema.tree.InMemoryDataTreeModification.resolveModificationStrategy(InMemoryDataTreeModification.java:137)<br/> at org.opendaylight.yangtools.yang.data.impl.schema.tree.InMemoryDataTreeModification.resolveModificationFor(InMemoryDataTreeModification.java:143)<br/> at org.opendaylight.yangtools.yang.data.impl.schema.tree.InMemoryDataTreeModification.merge(InMemoryDataTreeModification.java:73)<br/> at org.opendaylight.controller.md.sal.dom.store.impl.SnapshotBackedWriteTransaction.merge(SnapshotBackedWriteTransaction.java:85)<br/> at org.opendaylight.controller.cluster.datastore.modification.MergeModification.apply(MergeModification.java:37)<br/> at org.opendaylight.controller.cluster.datastore.modification.MutableCompositeModification.apply(MutableCompositeModification.java:33)<br/> at org.opendaylight.controller.cluster.datastore.Shard.commit(Shard.java:330)<br/> at org.opendaylight.controller.cluster.datastore.Shard.applyState(Shard.java:444)<br/> at org.opendaylight.controller.cluster.raft.RaftActor.onReceiveRecover(RaftActor.java:151)<br/> at org.opendaylight.controller.cluster.datastore.Shard.onReceiveRecover(Shard.java:175)<br/> at akka.persistence.UntypedPersistentActor$$anonfun$receiveRecover$1.applyOrElse(Eventsourced.scala:433)<br/> at scala.runtime.AbstractPartialFunction$mcVL$sp.apply$mcVL$sp(AbstractPartialFunction.scala:33)<br/> at scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:33)<br/> at scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:25)<br/> at akka.persistence.Eventsourced$$anonfun$akka$persistence$Eventsourced$$recoveryBehavior$1.applyOrElse(Eventsourced.scala:168)<br/> at akka.persistence.Recovery$State$$anonfun$processPersistent$1.apply(Recovery.scala:33)<br/> at akka.persistence.Recovery$State$$anonfun$processPersistent$1.apply(Recovery.scala:33)<br/> at akka.persistence.Recovery$class.withCurrentPersistent(Recovery.scala:176)<br/> at akka.persistence.UntypedPersistentActor.withCurrentPersistent(Eventsourced.scala:428)<br/> at akka.persistence.Recovery$State$class.processPersistent(Recovery.scala:33)<br/> at akka.persistence.Recovery$$anon$1.processPersistent(Recovery.scala:95)<br/> at akka.persistence.Recovery$$anon$1.aroundReceive(Recovery.scala:101)<br/> at akka.persistence.Recovery$class.aroundReceive(Recovery.scala:256)<br/> at akka.persistence.UntypedPersistentActor.akka$persistence$Eventsourced$$super$aroundReceive(Eventsourced.scala:428)<br/> at akka.persistence.Eventsourced$$anon$1.aroundReceive(Eventsourced.scala:35)<br/> at akka.persistence.Eventsourced$class.aroundReceive(Eventsourced.scala:369)<br/> at akka.persistence.UntypedPersistentActor.aroundReceive(Eventsourced.scala:428)<br/> at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)<br/> at akka.actor.ActorCell.invoke(ActorCell.scala:487)<br/> at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)<br/> at akka.dispatch.Mailbox.run(Mailbox.scala:220)<br/> at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393)<br/> at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)<br/> at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)<br/> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)<br/> at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)<br/> 2014-09-11 02:59:09,916 | WARN | lt-dispatcher-18 | OneForOneStrategy | 152 - com.typesafe.akka.slf4j - 2.3.4 | akka://opendaylight-cluster-data/user/shardmanager-config/member-3-shard-default-config | CacheLoader returned null for key (urn:opendaylight:params:xml:ns:yang:controller:config:sal-clustering-it:car-people?revision=2014-08-18)car-people.</p> <p>Operating System: Mac OS<br/> Platform: PC</p> CONTROLLER-808

Clustering : Exception on recovery Cache loader returning null for key

Bug Resolved Done Moiz Raja Moiz Raja Thu, 11 Sep 2014 14:48:58 +0000 Sat, 20 Sep 2014 18:29:43 +0000 Sat, 20 Sep 2014 18:29:43 +0000 Helium mdsal 0 5 <p>Schema for data you are trying to write is not loaded at time you are writing it.</p> <p>On recovery we try to read from the disk and apply the modification that we have recorded to the state which is in InMemoryDataStore. We start recovery only after the first SchemaContext update is received. It looks like if the schema context with the appropriate schema is not present when we are recovering this failure can occur. How do we get around this?</p> <p>Naive approach (not knowing what is in datastore):</p> <p>You need to store also list of YANG modules (namespaces) from schema context when snapshot was stored to disk and you could do full recovery (using inmemory data store) only when schema context contains all modules.</p> <p>More advanced approach is to know all used namespaces (stored data) and do recovery after all modules are present, but this requires bit more logic on<br/> writeout path and needs to store all used namespaces (even ones used in identities and instance identifiers stored in leaves).</p> <p>For persistence/recovery we are using akka-persistence. If we had to store schema related info using akka-persistence then we would need to recover first to get to the schema, which would also recover the data. </p> <p>To recover conditionally based on the presence of all required schema elements would therefore require us to go outside the bounds of akka-persistence.</p> <p>One possible solution I can think of is if schemaContext itself was serializable. If so that could be one proper alternative that we could use for recovery. Essentially it would work like this, when we receive a schemaContext during a normal run we would persist it to disk (using akka-persistence). On recovery we would load the schemaContext from disk and apply it to the in-memory store and then subsequently load the data. This seems like the most straightforward approach (if schemaContext was serializable that is)</p> <p>Let's say I had some data in the datastore for some module and later on I set a schema context on the datastore which did not contain a definition for that module. What would happen? Would it not be possible to retrieve that data from the datastore?</p> <p>I think the way SchemaContext works is problematic. At my previous company, we had a similar issue with failures caused by data model schemas not being present yet on restart so I preloaded all data model schemas from all bundles on restart by having a component that started first and scraped the schema-related files from all RESOLVED bundles. I think, longer term, we should do a similar thing with the SchemaContext so we don't have to keep putting in hacks like this.</p> <p>Tom, I agree in principle with your point - but for Helium we do need to hack this.</p> <p>Yeah I know - this should be investigated post-Helium.</p> <p>GlobalBundleScanningSchemaServiceImpl is one which exactly does that - gets schema sources from all resolved bundles.</p> <p>If I understand you correctly, what you are proposing is having component,<br/> which behaves as GlobalBundleScanningSchemaServiceImpl, but also caches schemas, so on subsequent restart (if nothing changed) it has all schemas<br/> ready at start?</p> <p>Actually this "hack" what I am proposing is also correct one, image some model bundles were unistalled, so still comparing schema context capabilities is still required (basicly because of restart and reinstall capabilities of your system changed). </p> <p>So yes, we need better heuristics how to recover datastore in various scenarios, which should be part of Lithium and also maybe Helium Service Release.</p> <p>Moiz's hack should work fine for now except if a module bundle is removed from the installation. On restart, the persisted modules set would contain a stale module and the modules in the new SchemaContext wouldn't "containAll" persisted modules so we wouldn't start the shards. But that would be uncommon so maybe we can live with it for now.</p> <p>It looks like it's the ModuleInfoBundleTracker that backs the SchemaContext. Currently it tracks ACTIVE bundles. I think if we changed it to track RESOLVED bundles, like GlobalBundleScanningSchemaServiceImpl does, it might fix this problem w/o having to do the hack. I wouldn't think there would be any adverse ramifications to this.</p> <p>What do you guys think?</p> <p>(In reply to Tony Tkacik from comment #9)<br/> &gt; GlobalBundleScanningSchemaServiceImpl is one which exactly does that - gets<br/> &gt; schema sources from all resolved bundles.<br/> &gt; <br/> &gt; If I understand you correctly, what you are proposing is having component,<br/> &gt; which behaves as GlobalBundleScanningSchemaServiceImpl, but also caches<br/> &gt; schemas, so on subsequent restart (if nothing changed) it has all schemas<br/> &gt; ready at start?<br/> &gt; <br/> &gt; Actually this "hack" what I am proposing is also correct one, image some<br/> &gt; model bundles were unistalled, so still comparing schema context<br/> &gt; capabilities is still required (basicly because of restart and reinstall<br/> &gt; capabilities of your system changed). <br/> &gt; <br/> &gt; So yes, we need better heuristics how to recover datastore in various<br/> &gt; scenarios, which should be part of Lithium and also maybe Helium Service<br/> &gt; Release.</p> <p>Tom, I'm not sure what would be the impact of that change - I wouldn't want to introduce it in Helium.</p> <p>From observation it does NOT seem that on startup schemaContext gets injected more than once most of the times. At exit however I see it being injected several times - I haven't investigated why yet.</p> <p>The config system starts modules async so many if not all bundles will have transitioned to ACTIVE at that point. But, of course, it depends on timing and is unpredictable. On shutdown, as module bundles are stopped, I believed they're removed from the SchemaContext - that would explain why you see updates. If so, that shouldn't screw up your changes because newModules wouldn't "containAll" knownModules so we wouldn't update the persistence.</p> <p>I understand your reservations about changing the module schema loading at this point. But it seems like it would be a benign change - I'd be interested in Tony's opinion. We can always change it post-Helium when there's more time to let it bake. Removing your "hack" should be simple. </p> <p>(In reply to Moiz Raja from comment #11)<br/> &gt; Tom, I'm not sure what would be the impact of that change - I wouldn't want<br/> &gt; to introduce it in Helium.<br/> &gt; <br/> &gt; From observation it does NOT seem that on startup schemaContext gets<br/> &gt; injected more than once most of the times. At exit however I see it being<br/> &gt; injected several times - I haven't investigated why yet.</p> <p><a href="https://git.opendaylight.org/gerrit/#/c/11346/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/11346/</a></p> Development External issue ID 1815 External issue URL ODL SR Target Milestone Priority Rank 0|i02mon: