https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/CONTROLLER-1650 controller <p>I'm seeing periodic SingleFeatureTest hangs in autorelease that seem to all trace back to this root issue:</p> <p>2017-05-01 10:44:13,976 | ERROR | rint Extender: 1 | BlueprintContainerImpl | 12 - org.apache.aries.blueprint.core - 1.7.1 | Unable to start blueprint container for bundle org.opendaylight.controller.sal-binding-broker-impl/1.5.0.Carbon due to unresolved dependencies <span class="error">&#91;(objectClass=org.opendaylight.mdsal.binding.generator.api.ClassLoadingStrategy)&#93;</span><br/> java.util.concurrent.TimeoutException<br/> at org.apache.aries.blueprint.container.BlueprintContainerImpl$1.run(BlueprintContainerImpl.java:371)<span class="error">&#91;12:org.apache.aries.blueprint.core:1.7.1&#93;</span></p> <p>I'm not sure what is causing this, but between this and BUG-8335, they amount for the <em>vast</em> majority of autorelease failures at the moment.</p> <p>Operating System: All<br/> Platform: All</p> CONTROLLER-1650

sal-binding-broker-impl times out waiting for ClassLoadingStrategy

Bug Resolved Done Robert Varga Colin Dixon Mon, 1 May 2017 20:34:29 +0000 Tue, 25 Jul 2023 08:24:33 +0000 Thu, 4 May 2017 09:32:06 +0000 blueprint 0 3 <p>Attachment org.opendaylight.odlparent.featuretest.SingleFeatureTest-output.txt.gz has been added with description: this is an example of the failure, takeng from failnever job #3</p> <p>Jobs that failed with SFT hangs:<br/> <a href="https://jenkins.opendaylight.org/releng/view/autorelease/job/autorelease-release-carbon/271/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/autorelease/job/autorelease-release-carbon/271/</a><br/> <a href="https://jenkins.opendaylight.org/releng/view/autorelease/job/autorelease-release-failnever-carbon/1/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/autorelease/job/autorelease-release-failnever-carbon/1/</a><br/> <a href="https://jenkins.opendaylight.org/releng/view/autorelease/job/autorelease-release-failnever-carbon/3/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/autorelease/job/autorelease-release-failnever-carbon/3/</a><br/> <a href="https://jenkins.opendaylight.org/releng/view/autorelease/job/autorelease-release-failnever-carbon/4/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/autorelease/job/autorelease-release-failnever-carbon/4/</a></p> <p>Robert Varga thinks this is the issue:</p> <p>2017-05-01 06:14:00,894 | WARN | pool-2-thread-1 | AugmentStatementImpl | 370 - org.opendaylight.yangtools.yang-parser-impl - 1.1.0.Carbon | Failed to add augmentation alto-model-config:63:12 defined at alto-manual-maps:80:4<br/> org.opendaylight.yangtools.yang.parser.spi.meta.InferenceException: An augment cannot add node 'resource-id' because it is mandatory and in module different than target <span class="error">&#91;at alto-networkmap-config:49:8&#93;</span><br/> at org.opendaylight.yangtools.yang.parser.spi.meta.InferenceException.throwIf(InferenceException.java:47)<br/> at org.opendaylight.yangtools.yang.parser.stmt.rfc6020.AugmentStatementImpl$Definition.checkForMandatoryNodes(AugmentStatementImpl.java:301)<br/> at java.util.ArrayList.forEach(ArrayList.java:1249)<span class="error">&#91;:1.8.0_121&#93;</span><br/> at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)<span class="error">&#91;:1.8.0_121&#93;</span><br/> at org.opendaylight.yangtools.yang.parser.stmt.rfc6020.AugmentStatementImpl$Definition.checkForMandatoryNodes(AugmentStatementImpl.java:298)<br/> at org.opendaylight.yangtools.yang</p> <p>It appears when the feature service is starting config-manager in the recent alto failure. The result is that config-manager is never started and thus the ClassLoadingStrategy is never exported.</p> <p>This doesn't happen in the NEMO case, but in the NEMO case it seems like for whatever reason there's a deadlock if the ClassLoadingStrategy isn't exported.</p> <p>The problems seem to be related to the fact config-manager either not being started (in the nemo case) or it seems to throw and exception (alto case).</p> <p>In either case it seems config-manager's activator is not completing.</p> <p>It sounds like there might be a way to change the run level of the config-manager to make sure it loads first or early.</p> <p>Vratko pointed out that the last failures in carbon-merge:<br/> <a href="https://jenkins.opendaylight.org/releng/view/Merge-Carbon/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/Merge-Carbon/</a></p> <p>Is a reasonable place to look for if this occurs outside of autorelease.</p> <p>For what it's worth, from the lastFailedBuilds, 3 of the hung in SFT:<br/> <a href="https://jenkins.opendaylight.org/releng/view/Merge-Carbon/job/distribution-merge-carbon/1717/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/Merge-Carbon/job/distribution-merge-carbon/1717/</a><br/> <a href="https://jenkins.opendaylight.org/releng/view/Merge-Carbon/job/netvirt-merge-carbon/737/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/Merge-Carbon/job/netvirt-merge-carbon/737/</a><br/> <a href="https://jenkins.opendaylight.org/releng/view/Merge-Carbon/job/netconf-merge-carbon/257/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/Merge-Carbon/job/netconf-merge-carbon/257/</a></p> <p>At least the distribution issue appears to be an occurrence of the same thing, so it's not just an autorelease issue:<br/> odl-integration-all in distribution merge carbon 1717:<br/> <a href="https://logs.opendaylight.org/releng/jenkins092/distribution-merge-carbon/1717/archives/features/singles/odl-integration-all/target/surefire-reports/org.opendaylight.odlparent.featuretest.SingleFeatureTest-output.txt.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/distribution-merge-carbon/1717/archives/features/singles/odl-integration-all/target/surefire-reports/org.opendaylight.odlparent.featuretest.SingleFeatureTest-output.txt.gz</a></p> <p>2017-04-30 11:42:25,477 | ERROR | rint Extender: 3 | BlueprintContainerImpl | 12 - org.apache.aries.blueprint.core - 1.7.1 | Unable to start blueprint container for bundle org.opendaylight.controller.sal-binding-broker-impl/1.5.0.SNAPSHOT due to unresolved dependencies <span class="error">&#91;(objectClass=org.opendaylight.mdsal.binding.generator.api.ClassLoadingStrategy)&#93;</span></p> <p>The netvirt case doesn't seem have surefire logs for the odl-netvirt-openstack feature.</p> <p>The netconf case doesn't end up hitting exactly the same issue, but it's similar:<br/> <a href="https://logs.opendaylight.org/releng/jenkins092/netconf-merge-carbon/257/archives/features/netconf-connector/odl-message-bus/target/surefire-reports/org.opendaylight.odlparent.featuretest.SingleFeatureTest-output.txt.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/netconf-merge-carbon/257/archives/features/netconf-connector/odl-message-bus/target/surefire-reports/org.opendaylight.odlparent.featuretest.SingleFeatureTest-output.txt.gz</a></p> <p>2017-04-28 14:11:47,635 | ERROR | rint Extender: 2 | BlueprintContainerImpl | 12 - org.apache.aries.blueprint.core - 1.7.1 | Unable to start blueprint container for bundle org.opendaylight.netconf.messagebus-netconf/1.2.0.SNAPSHOT due to unresolved dependencies <span class="error">&#91;(objectClass=org.opendaylight.controller.messagebus.spi.EventSourceRegistry)&#93;</span></p> <p>We should add logging in ConfigManagerActivator to see if it even runs. I can push a patch for that. It already catches and logs Exception (WARN) and Error (ERROR) so we should know if it failed.</p> <p>If it didn't even run, that indicates either bundle didn't resolve (unlikely) or some other bundle activator is stuck preventing ConfigManagerActivator from running. For the latter we'd need a thread dump.</p> <p><a href="https://git.opendaylight.org/gerrit/#/c/56406/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/56406/</a> to add info logging.</p> <p>carbon: <a href="https://git.opendaylight.org/gerrit/56437" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/56437</a></p> Blocks RELENG-47 Development External issue ID 8342 External issue URL Issue Type ODL SR Target Milestone Rank 0|i02rvr: