https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/MDSAL-119 mdsal <p>AAA is using MDSAL DataBroker for clustering the AAA user/pass/token data. Since we moved to MDSAL we are getting a lot of complain from different users that their unable to login. After investigation, we found that the root cause of this is that the cluster data broker isn't loading properly due to the following exception:</p> <p>SchemaResolutionException{unsatisfiedImports={SourceIdentifier <span class="error">&#91;name=opendaylight-flow-table-statistics@2013-12-15&#93;</span>=[ModuleImportImpl [n... (full log attached)</p> <p>Due to that, AAA transactions are timing out and the user is unable to login or get a token.</p> <p>Operating System: All<br/> Platform: All</p> MDSAL-119

Sometimes getting "SchemaResolutionException" exception when starting controller.

Bug Resolved Cannot Reproduce Unassigned Sharon Aicler Wed, 2 Dec 2015 23:00:07 +0000 Fri, 9 Mar 2018 18:00:11 +0000 Fri, 26 Jan 2018 16:05:34 +0000 0 6 <p>Attachment karaf.zip has been added with description: The log with the exception in it + showing that AAA transactions are timing out.</p> <p>extract from logs:</p> <p>2015-11-24 20:16:05,209 | INFO | l for user karaf | YangTextSchemaContextResolver | 78 - org.opendaylight.yangtools.yang-parser-impl - 0.8.0.SNAPSHOT | Failed to fully assemble schema context for [SourceIdentifier <span class="error">&#91;name=yang-ext@2013-07-09&#93;</span>, SourceIdentifier <span class="error">&#91;name=iana-afn-safi@2013-07-04&#93;</span>, SourceIdentifier <span class="error">&#91;name=iana-if-type@2014-05-08&#93;</span>, SourceIdentifier <span class="error">&#91;name=ietf-inet-types@2010-09-24&#93;</span>, SourceIdentifier <span class="error">&#91;name=ietf-interfaces@2014-05-08&#93;</span>, SourceIdentifier <span class="error">&#91;name=ietf-yang-types@2010-09-24&#93;</span>, SourceIdentifier <span class="error">&#91;name=ietf-yang-types@2013-07-15&#93;</span>, SourceIdentifier <span class="error">&#91;name=ietf-restconf@2013-10-19&#93;</span>, SourceIdentifier <span class="error">&#91;name=opendaylight-l2-types@2013-08-27&#93;</span>, SourceIdentifier <span class="error">&#91;name=ted@2013-07-12&#93;</span>, SourceIdentifier <span class="error">&#91;name=ted@2013-10-21&#93;</span>, SourceIdentifier <span class="error">&#91;name=network-topology@2013-07-12&#93;</span>, SourceIdentifier <span class="error">&#91;name=network-topology@2013-10-21&#93;</span>, SourceIdentifier <span class="error">&#91;name=isis-topology@2013-07-12&#93;</span>, SourceIdentifier <span class="error">&#91;name=isis-topology@2013-10-21&#93;</span>, SourceIdentifier <span class="error">&#91;name=ospf-topology@2013-07-12&#93;</span>, SourceIdentifier <span class="error">&#91;name=ospf-topology@2013-10-21&#93;</span>, SourceIdentifier <span class="error">&#91;name=l3-unicast-igp-topology@2013-07-12&#93;</span>, SourceIdentifier <span class="error">&#91;name=l3-unicast-igp-topology@2013-10-21&#93;</span>, SourceIdentifier <span class="error">&#91;name=ietf-network-topology@2015-06-08&#93;</span>, SourceIdentifier <span class="error">&#91;name=ietf-network@2015-06-08&#93;</span>, SourceIdentifier <span class="error">&#91;name=ietf-inet-types@2013-07-15&#93;</span>, SourceIdentifier <span class="error">&#91;name=config@2013-04-05&#93;</span>, SourceIdentifier <span class="error">&#91;name=rpc-context@2013-06-17&#93;</span>, SourceIdentifier <span class="error">&#91;name=general-entity@2015-08-20&#93;</span>, SourceIdentifier <span class="error">&#91;name=opendaylight-entity-ownership-service@2015-08-10&#93;</span>, SourceIdentifier <span class="error">&#91;name=shutdown@2013-12-18&#93;</span>, SourceIdentifier <span class="error">&#91;name=shutdown-impl@2013-12-18&#93;</span>, SourceIdentifier <span class="error">&#91;name=netty@2013-11-19&#93;</span>, SourceIdentifier <span class="error">&#91;name=netty-event-executor@2013-11-12&#93;</span>, SourceIdentifier <span class="error">&#91;name=threadgroup@2013-11-07&#93;</span>, SourceIdentifier <span class="error">&#91;name=netty-timer@2013-11-19&#93;</span>, SourceIdentifier <span class="error">&#91;name=threadpool@2013-04-09&#93;</span>, SourceIdentifier <span class="error">&#91;name=threadpool-impl-fixed@2013-12-01&#93;</span>, SourceIdentifier <span class="error">&#91;name=threadpool-impl-flexible@2013-12-01&#93;</span>, SourceIdentifier <span class="error">&#91;name=threadpool-impl-scheduled@2013-12-01&#93;</span>, SourceIdentifier <span class="error">&#91;name=threadpool-impl@2013-04-05&#93;</span>, SourceIdentifier <span class="error">&#91;name=opendaylight-config-dom-datastore@2014-06-17&#93;</span>, SourceIdentifier <span class="error">&#91;name=opendaylight-md-sal-common@2013-10-28&#93;</span>, SourceIdentifier <span class="error">&#91;name=opendaylight-md-sal-dom@2013-10-28&#93;</span>, SourceIdentifier <span class="error">&#91;name=opendaylight-operational-dom-datastore@2014-06-17&#93;</span>, SourceIdentifier <span class="error">&#91;name=opendaylight-sal-binding-broker-impl@2013-10-28&#93;</span>, SourceIdentifier <span class="error">&#91;name=opendaylight-md-sal-binding@2013-10-28&#93;</span>, SourceIdentifier <span class="error">&#91;name=opendaylight-inmemory-datastore-provider@2014-06-17&#93;</span>, SourceIdentifier <span class="error">&#91;name=opendaylight-sal-dom-broker-impl@2013-10-28&#93;</span>, SourceIdentifier <span class="error">&#91;name=opendaylight-pingpong-broker@2014-11-07&#93;</span>, SourceIdentifier <span class="error">&#91;name=actor-system-provider-service@2015-10-05&#93;</span>, SourceIdentifier <span class="error">&#91;name=remote-rpc-connector@2014-07-07&#93;</span>, SourceIdentifier <span class="error">&#91;name=actor-system-provider-impl@2015-10-05&#93;</span>, SourceIdentifier <span class="error">&#91;name=cluster-admin-provider@2015-10-13&#93;</span>, SourceIdentifier <span class="error">&#91;name=cluster-admin@2015-10-13&#93;</span>, SourceIdentifier <span class="error">&#91;name=distributed-datastore-provider@2014-06-12&#93;</span>, SourceIdentifier <span class="error">&#91;name=distributed-entity-ownership-service@2015-08-10&#93;</span>, SourceIdentifier <span class="error">&#91;name=entity-owners@2015-08-04&#93;</span>, SourceIdentifier <span class="error">&#91;name=odl-concurrent-data-broker-cfg@2014-11-24&#93;</span>, SourceIdentifier <span class="error">&#91;name=netconf-node-inventory@2014-01-08&#93;</span>, SourceIdentifier <span class="error">&#91;name=opendaylight-inventory@2013-08-19&#93;</span>, SourceIdentifier <span class="error">&#91;name=opendaylight-topology-inventory@2013-10-30&#93;</span>, SourceIdentifier <span class="error">&#91;name=opendaylight-topology-view@2013-10-30&#93;</span>, SourceIdentifier <span class="error">&#91;name=opendaylight-topology@2013-10-30&#93;</span>, SourceIdentifier <span class="error">&#91;name=opendaylight-action-types@2013-11-12&#93;</span>, SourceIdentifier <span class="error">&#91;name=opendaylight-flow-types@2013-10-26&#93;</span>, SourceIdentifier <span class="error">&#91;name=opendaylight-group-types@2013-10-18&#93;</span>, SourceIdentifier <span class="error">&#91;name=opendaylight-match-types@2013-10-26&#93;</span>, SourceIdentifier <span class="error">&#91;name=opendaylight-meter-types@2013-09-18&#93;</span>, SourceIdentifier <span class="error">&#91;name=opendaylight-port-types@2013-09-25&#93;</span>, SourceIdentifier <span class="error">&#91;name=opendaylight-queue-types@2013-09-25&#93;</span>, SourceIdentifier <span class="error">&#91;name=opendaylight-table-types@2013-10-26&#93;</span>, SourceIdentifier <span class="error">&#91;name=opendaylight-flow-statistics@2013-08-19&#93;</span>, SourceIdentifier <span class="error">&#91;name=opendaylight-flow-table-statistics@2013-12-15&#93;</span>, SourceIdentifier <span class="error">&#91;name=opendaylight-group-statistics@2013-11-11&#93;</span>, SourceIdentifier <span class="error">&#91;name=opendaylight-meter-statistics@2013-11-11&#93;</span>, SourceIdentifier <span class="error">&#91;name=opendaylight-port-statistics@2013-12-14&#93;</span>, SourceIdentifier <span class="error">&#91;name=opendaylight-queue-statistics@2013-12-16&#93;</span>, SourceIdentifier <span class="error">&#91;name=opendaylight-statistics-types@2013-09-25&#93;</span>]<br/> SchemaResolutionException{unsatisfiedImports={SourceIdentifier <span class="error">&#91;name=opendaylight-flow-table-statistics@2013-12-15&#93;</span>=[ModuleImportImpl <span class="error">&#91;name=flow-node-inventory, revision=2013-08-19&#93;</span>, ModuleImportImpl <span class="error">&#91;name=flow-capable-transaction, revision=2015-03-04&#93;</span>], SourceIdentifier <span class="error">&#91;name=opendaylight-flow-statistics@2013-08-19&#93;</span>=[ModuleImportImpl <span class="error">&#91;name=flow-capable-transaction, revision=null&#93;</span>, ModuleImportImpl <span class="error">&#91;name=flow-node-inventory, revision=2013-08-19&#93;</span>], SourceIdentifier <span class="error">&#91;name=opendaylight-port-statistics@2013-12-14&#93;</span>=[ModuleImportImpl <span class="error">&#91;name=flow-capable-transaction, revision=2015-03-04&#93;</span>], SourceIdentifier <span class="error">&#91;name=opendaylight-queue-statistics@2013-12-16&#93;</span>=[ModuleImportImpl <span class="error">&#91;name=flow-node-inventory, revision=2013-08-19&#93;</span>, ModuleImportImpl <span class="error">&#91;name=flow-capable-transaction, revision=2015-03-04&#93;</span>], SourceIdentifier <span class="error">&#91;name=opendaylight-meter-statistics@2013-11-11&#93;</span>=[ModuleImportImpl <span class="error">&#91;name=flow-node-inventory, revision=2013-08-19&#93;</span>, ModuleImportImpl <span class="error">&#91;name=flow-capable-transaction, revision=2015-03-04&#93;</span>], SourceIdentifier <span class="error">&#91;name=opendaylight-group-statistics@2013-11-11&#93;</span>=[ModuleImportImpl <span class="error">&#91;name=flow-node-inventory, revision=2013-08-19&#93;</span>, ModuleImportImpl <span class="error">&#91;name=flow-capable-transaction, revision=2015-03-04&#93;</span>]}}<br/> at org.opendaylight.yangtools.yang.parser.repo.SharedSchemaContextFactory$2.apply(SharedSchemaContextFactory.java:77)<br/> at org.opendaylight.yangtools.yang.parser.repo.SharedSchemaContextFactory$2.apply(SharedSchemaContextFactory.java:64)<br/> at com.google.common.util.concurrent.Futures$ChainingListenableFuture.run(Futures.java:906)<br/> at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)<br/> at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)<br/> at com.google.common.util.concurrent.ExecutionList.add(ExecutionList.java:101)<br/> at com.google.common.util.concurrent.AbstractFuture.addListener(AbstractFuture.java:170)<br/> at com.google.common.util.concurrent.Futures.transform(Futures.java:567)<br/> at org.opendaylight.yangtools.yang.parser.repo.SharedSchemaContextFactory.createSchemaContext(SharedSchemaContextFactory.java:136)<br/> at org.opendaylight.yangtools.yang.parser.repo.YangTextSchemaContextResolver.getSchemaContext(YangTextSchemaContextResolver.java:204)<br/> at org.opendaylight.yangtools.yang.parser.repo.URLSchemaContextResolver.getSchemaContext(URLSchemaContextResolver.java:63)<br/> at org.opendaylight.controller.sal.dom.broker.GlobalBundleScanningSchemaServiceImpl.tryToUpdateSchemaContext(GlobalBundleScanningSchemaServiceImpl.java:251)<br/> at org.opendaylight.controller.sal.dom.broker.GlobalBundleScanningSchemaServiceImpl$BundleScanner.addingBundle(GlobalBundleScanningSchemaServiceImpl.java:199)<br/> at org.opendaylight.controller.sal.dom.broker.GlobalBundleScanningSchemaServiceImpl$BundleScanner.addingBundle(GlobalBundleScanningSchemaServiceImpl.java:172)<br/> at org.osgi.util.tracker.BundleTracker$Tracked.customizerAdding(BundleTracker.java:467)<span class="error">&#91;karaf-org.osgi.core.jar:&#93;</span><br/> at org.osgi.util.tracker.BundleTracker$Tracked.customizerAdding(BundleTracker.java:414)<span class="error">&#91;karaf-org.osgi.core.jar:&#93;</span><br/> at org.osgi.util.tracker.AbstractTracked.trackAdding(AbstractTracked.java:256)<span class="error">&#91;karaf-org.osgi.core.jar:&#93;</span><br/> at org.osgi.util.tracker.AbstractTracked.track(AbstractTracked.java:229)<span class="error">&#91;karaf-org.osgi.core.jar:&#93;</span><br/> at org.osgi.util.tracker.BundleTracker$Tracked.bundleChanged(BundleTracker.java:443)<span class="error">&#91;karaf-org.osgi.core.jar:&#93;</span><br/> at org.eclipse.osgi.framework.internal.core.BundleContextImpl.dispatchEvent(BundleContextImpl.java:847)<br/> at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:230)<br/> at org.eclipse.osgi.framework.eventmgr.ListenerQueue.dispatchEventSynchronous(ListenerQueue.java:148)</p> <p>Decreased to normal, since this is known behaviour since Hydrogen:</p> <p>What actually is happening OSGI framework is reporting bundles and models<br/> in random order, even if they have clear depenency resolution hierarchy,<br/> eg. it is common for underyling OSGI Framework to report<br/> every model except ietf-yang-types, which is base for all them,<br/> that means MD-SAL does several cycles till OSGI announce that model.</p> <p>Unfortunatelly we can not affect the order in which OSGI is reporting bundles (and thus models), so sometimes startup takes longer.</p> <p>Lithium <a href="https://jira.opendaylight.org/browse/CONTROLLER-1458" title="config-pusher: Unable to push configuration due to missing yang models" class="issue-link" data-issue-key="CONTROLLER-1458"><del>CONTROLLER-1458</del></a> is similar to this.</p> <p>&gt; sometimes startup takes longer.</p> <p>I am not sure about this bug, but <a href="https://jira.opendaylight.org/browse/CONTROLLER-1458" title="config-pusher: Unable to push configuration due to missing yang models" class="issue-link" data-issue-key="CONTROLLER-1458"><del>CONTROLLER-1458</del></a> states "longer" is more than 10 minutes.</p> <p><a href="https://jira.opendaylight.org/browse/AAA-72" title="AAA StoreBuilder init times out too early" class="issue-link" data-issue-key="AAA-72"><del>AAA-72</del></a> if marked as fixed. Does that fix this Bug too?</p> <p>No... <a href="https://jira.opendaylight.org/browse/AAA-72" title="AAA StoreBuilder init times out too early" class="issue-link" data-issue-key="AAA-72"><del>AAA-72</del></a> was causing lack of initialization of AAA data store (Resulting in 401 exceptions). This bug is separate.</p> <p>So we are activating via a plain old activator/ServiceTracker. I think this could be solved by activating AAA via config subsystem &#8211; where the activation can wait until a specific model is available in the global SchemaContext.</p> <p>E.g. in the config xml file specify the capability corresponding to the model required for proper operation and then the config pusher will know to wait.</p> <p>We have applied this workaround a couple of times since Helium, if I remember correctly.</p> <p>Would that work or is OSGi activation strictly required?</p> <p>What was the sequence that led up to this? From the stack trace, the resolution error occurs after a bundleChanged event which tells me that this was a result of doing a manual feature:install. Is that correct? </p> <p>Since the GlobalBundleScanningSchemaServiceImpl scans resolved bundles this shouldn't happen on a restart or if featuresBoot is used b/c it gets all bundles at once and all yang models should be available when GlobalBundleScanningSchemaServiceImpl is activated and tries to build the schema context he first time.</p> <p>Also, even on feature:install, all new bundles have to be resolved before it can start activating them. So even if the GlobalBundleScanningSchemaServiceImpl failed initially b/c it hadn't gotten notified of a dependent yang model bundle yet, it should eventually get all the bundle change events and succeed. At least that's what I think - I may be missing something. I would suggest turning on GlobalBundleScanningSchemaServiceImpl debug to see if that's the case. </p> <p>Does the clustered data store fail to initialize and is rendered useless b/c of this? Or is it just AAA that is affected? If the former then this is a bigger problem and the rest of the system is hosed so it doesn't really matter that AAA can't authenticate and not using md-sal in AAA isn't really a solution. If the latter then Robert's suggestion sounds like it should work.</p> <p>(In reply to Tom Pantelis from comment #7)<br/> &gt; What was the sequence that led up to this? From the stack trace, the<br/> &gt; resolution error occurs after a bundleChanged event which tells me that this<br/> &gt; was a result of doing a manual feature:install. Is that correct? <br/> &gt; <br/> &gt; Since the GlobalBundleScanningSchemaServiceImpl scans resolved bundles this<br/> &gt; shouldn't happen on a restart or if featuresBoot is used b/c it gets all<br/> &gt; bundles at once and all yang models should be available when<br/> &gt; GlobalBundleScanningSchemaServiceImpl is activated and tries to build the<br/> &gt; schema context he first time.</p> <p>Cannot reproduce this anymore.</p> Development External issue ID 4702 External issue URL Priority Rank 0|i02wfj: