https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/NETCONF-358 netconf <p>Recent merge job, while testing features4-netconf-connector, the SingleFeaturesTest passed, but there are many ERRORs on the log <span class="error">&#91;0&#93;</span>.</p> <p>I suspect this causes failure <span class="error">&#91;1&#93;</span> in Integration/Distribution karaf patch.</p> <p><span class="error">&#91;0&#93;</span> <a href="https://logs.opendaylight.org/releng/jenkins092/netconf-merge-carbon/193/archives/features/netconf-connector/features4-netconf-connector/target/surefire-reports/org.opendaylight.odlparent.featuretest.SingleFeatureTest-output.txt.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/netconf-merge-carbon/193/archives/features/netconf-connector/features4-netconf-connector/target/surefire-reports/org.opendaylight.odlparent.featuretest.SingleFeatureTest-output.txt.gz</a><br/> <span class="error">&#91;1&#93;</span> <a href="https://jenkins.opendaylight.org/releng/job/distribution-verify-carbon/org.opendaylight.integration$features-index/37/testReport/junit/installFeature(org.opendaylight.odlparent.featuretest.SingleFeatureTest)[repoUrl_%20file__w_workspace_distribution-verify-carbon_features_repos_index_target_feature_feature.xml,%20Feature_%20features4-netconf-connector%201.2.0/SNAPSHOT]/installFeature/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/job/distribution-verify-carbon/org.opendaylight.integration$features-index/37/testReport/junit/installFeature(org.opendaylight.odlparent.featuretest.SingleFeatureTest)[repoUrl_%20file__w_workspace_distribution-verify-carbon_features_repos_index_target_feature_feature.xml,%20Feature_%20features4-netconf-connector%201.2.0/SNAPSHOT]/installFeature/</a></p> <p>Operating System: All<br/> Platform: All</p> NETCONF-358

Fix failures in Karaf 4 feature tests

Bug Resolved Done Jakub Morvay Vratko Polak Thu, 2 Mar 2017 18:27:32 +0000 Fri, 15 Mar 2019 22:22:36 +0000 Mon, 31 Jul 2017 10:43:11 +0000 netconf 0 8 <p>Has this been fixed? Do we know?</p> <p>Currently we do not know, the feature repos are still commented-out from Karaf 4 feature-index.</p> <p>Oh, I have not realized that this is visible even without being tested by distribution-check.</p> <p>For example, considering features4-netconf-connector, in recent surefire log <span class="error">&#91;2&#93;</span> I see "Transaction failed on 2PC, server is unhealthy", which would fail distribution-check.</p> <p>But I believe most of the errors come <span class="error">&#91;3&#93;</span> from features4-config-netty of Controller project.<br/> After fixing odl-config-netty, other faulty dependencies might become visible.</p> <p><span class="error">&#91;2&#93;</span> <a href="https://logs.opendaylight.org/releng/jenkins092/netconf-merge-carbon/248/archives/features/netconf-connector/features4-netconf-connector/target/surefire-reports/org.opendaylight.odlparent.featuretest.SingleFeatureTest-output.txt.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/netconf-merge-carbon/248/archives/features/netconf-connector/features4-netconf-connector/target/surefire-reports/org.opendaylight.odlparent.featuretest.SingleFeatureTest-output.txt.gz</a><br/> <span class="error">&#91;3&#93;</span> <a href="https://logs.opendaylight.org/releng/jenkins092/controller-merge-carbon/428/archives/features/config-netty/features4-config-netty/target/surefire-reports/org.opendaylight.odlparent.featuretest.SingleFeatureTest-output.txt.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/controller-merge-carbon/428/archives/features/config-netty/features4-config-netty/target/surefire-reports/org.opendaylight.odlparent.featuretest.SingleFeatureTest-output.txt.gz</a></p> <p>&gt; distribution-check</p> <p>To be more specific: </p> {project}-distribution4-check-carbon would be the job to prevent {project} <p> from messing up features already included in Karaf 4 odl-integration-all.<br/> distribution4-check-carbon is the job which should reject adding faulty features to Karaf 4 odl-integration-all.</p> <p>"Transaction failed on 2PC, server is unhealthy" is caused by restart during feature install. Restart is triggered, when odl-mdsal-broker-local is installed in netconf karaf distribution.<br/> However, after this restart, features seem to be working well.</p> <p>Is this fixed?</p> <p>Testing this was limited by slow distribution4-check jobs.<br/> We will know more when this <span class="error">&#91;4&#93;</span> one finishes.</p> <p><span class="error">&#91;4&#93;</span> <a href="https://jenkins.opendaylight.org/releng/job/distribution4-check-carbon/21" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/job/distribution4-check-carbon/21</a></p> <p>Actually it is worse. After <a href="https://jira.opendaylight.org/browse/ODLPARENT-75" title="Feature repositories should aggregate (as opposed to redefine) single features." class="issue-link" data-issue-key="ODLPARENT-75"><del>ODLPARENT-75</del></a> netconf merge job shows SFT failures on multiple features. Similar to <a href="https://jira.opendaylight.org/browse/NETCONF-411" title="odl-netconf-console fails SFT due to callhomeProvider" class="issue-link" data-issue-key="NETCONF-411"><del>NETCONF-411</del></a>.</p> <p>The good news is that netconf devs now know what exactly is missing, so they can start fixing it.</p> <p>Good news:</p> <ul> <li>Netconf merge job is passing.</li> <li>Netconf distribution4-check is passing (but no non-restconf netconf feature in distribution).</li> <li>Change to add netconf feature repos <span class="error">&#91;5&#93;</span> passed its verify jobs.</li> </ul> <p>Bad news:</p> <ul> <li>I still see errors in surefire reports. <ul> <li>For example "server is unhealthy" <span class="error">&#91;6&#93;</span> would very probably fail distribution4-check if the feature is added to odl-integration-all.</li> </ul> </li> </ul> <p><span class="error">&#91;5&#93;</span> <a href="https://git.opendaylight.org/gerrit/56036" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/56036</a><br/> <span class="error">&#91;6&#93;</span> <a href="https://logs.opendaylight.org/releng/jenkins092/netconf-merge-carbon/254/archives/features/netconf-connector/odl-netconf-clustered-topology/target/surefire-reports/org.opendaylight.odlparent.featuretest.SingleFeatureTest-output.txt.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/netconf-merge-carbon/254/archives/features/netconf-connector/odl-netconf-clustered-topology/target/surefire-reports/org.opendaylight.odlparent.featuretest.SingleFeatureTest-output.txt.gz</a></p> <p>I've added netconf features to odl-integration all <span class="error">&#91;0&#93;</span>. Distribution4-check <span class="error">&#91;1&#93;</span> failed with following error, which is probably unrelated to restarts mentioned above.</p> <p>2017-04-28 12:37:01,523 | ERROR | pool-1-thread-2 | BootFeaturesInstaller | 7 - org.apache.karaf.features.core - 4.0.9 | Error installing boot features<br/> org.osgi.service.resolver.ResolutionException: Unable to resolve root: missing requirement <span class="error">&#91;root&#93;</span> osgi.identity; osgi.identity=odl-netconf-tcp; type=karaf.feature; version=1.2.0.SNAPSHOT; filter:="(&amp;(osgi.identity=odl-netconf-tcp)(type=karaf.feature)(version&gt;=1.2.0.SNAPSHOT))" [caused by: Unable to resolve odl-netconf-tcp/1.2.0.SNAPSHOT: missing requirement <span class="error">&#91;odl-netconf-tcp/1.2.0.SNAPSHOT&#93;</span> osgi.identity; osgi.identity=odl-netconf-impl; type=karaf.feature; version="<span class="error">&#91;1.2.0.SNAPSHOT,1.2.0.SNAPSHOT&#93;</span>" [caused by: Unable to resolve odl-netconf-impl/1.2.0.SNAPSHOT: missing requirement <span class="error">&#91;odl-netconf-impl/1.2.0.SNAPSHOT&#93;</span> osgi.identity; osgi.identity=odl-netconf-netty-util; type=karaf.feature; version="<span class="error">&#91;1.2.0.SNAPSHOT,1.2.0.SNAPSHOT&#93;</span>" [caused by: Unable to resolve odl-netconf-netty-util/1.2.0.SNAPSHOT: missing requirement <span class="error">&#91;odl-netconf-netty-util/1.2.0.SNAPSHOT&#93;</span> osgi.identity; osgi.identity=org.opendaylight.netconf.netty-util; type=osgi.bundle; version="<span class="error">&#91;1.2.0.SNAPSHOT,1.2.0.SNAPSHOT&#93;</span>"; resolution:=mandatory [caused by: Unable to resolve org.opendaylight.netconf.netty-util/1.2.0.SNAPSHOT: missing requirement <span class="error">&#91;org.opendaylight.netconf.netty-util/1.2.0.SNAPSHOT&#93;</span> osgi.wiring.package; filter:="(&amp;(osgi.wiring.package=org.apache.sshd)(version&gt;=0.14.0)(!(version&gt;=1.0.0)))"]]]]</p> <p><span class="error">&#91;0&#93;</span> <a href="https://git.opendaylight.org/gerrit/#/c/56036/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/56036/</a><br/> <span class="error">&#91;1&#93;</span> <a href="https://logs.opendaylight.org/releng/jenkins092/distribution4-check-carbon/31/console.log.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/distribution4-check-carbon/31/console.log.gz</a></p> <p>Andrej has created a patch <span class="error">&#91;7&#93;</span>, which dances around exclusion of sshd-core.<br/> So perhaps making sure excluded artifacts are installed in "empty" karaf is needed: <span class="error">&#91;8&#93;</span>.</p> <p><span class="error">&#91;7&#93;</span> <a href="https://git.opendaylight.org/gerrit/56255" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/56255</a><br/> <span class="error">&#91;8&#93;</span> <a href="https://git.opendaylight.org/gerrit/56262" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/56262</a></p> <p>ResolutionException is thrown when netconf feature is placed to featuresBoot in org.apache.karaf.features.cfg regardless of presence ssh feature. Start via console feature:install works though.</p> <p><a href="https://git.opendaylight.org/gerrit/#/c/56421/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/56421/</a></p> <p>That patch isn't passing verify:</p> <p>org.opendaylight.odlparent.featuretest.SingleFeatureTest<br/> 10:22:18 installFeatureCatchAndLog(org.opendaylight.odlparent.featuretest.SingleFeatureTest)<span class="error">&#91;repoUrl: file:/w/workspace/netconf-verify-nitrogen-mvn33-openjdk8/features/netconf-connector/features4-netconf-connector/target/feature/feature.xml, Feature: features4-netconf-connector 1.3.0.SNAPSHOT&#93;</span> Time elapsed: 58.96 sec &lt;&lt;&lt; ERROR!<br/> 10:22:18 org.opendaylight.yangtools.yang.data.api.schema.tree.ConflictingModificationAppliedException: Node was replaced by other transaction.<br/> 10:22:18 at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkConflicting(SchemaAwareApplyOperation.java:78)</p> <p>carbon: <a href="https://git.opendaylight.org/gerrit/#/c/56460/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/56460/</a></p> <p>Carbon fix was merged. However verification is blocked by <a href="https://jira.opendaylight.org/browse/AAA-126" title="Unable to start blueprint container for bundle org.opendaylight.aaa.idmlight" class="issue-link" data-issue-key="AAA-126"><del>AAA-126</del></a> since it causes failures in distribution4-check.<br/> <a href="https://jenkins.opendaylight.org/releng/job/distribution4-check-carbon/66/console" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/job/distribution4-check-carbon/66/console</a></p> <p>ResolutionException is not present in distribution4-check logs<span class="error">&#91;9&#93;</span> anymore after merging <span class="error">&#91;10&#93;</span>. Test still fails on "server is unhealthy" check, which is caused by restarting config bundles. Bundle restarting is discussed on Release mailing list<span class="error">&#91;11&#93;</span>.</p> <p><span class="error">&#91;9&#93;</span> <a href="https://jenkins.opendaylight.org/releng/job/distribution4-check-carbon/72/console" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/job/distribution4-check-carbon/72/console</a><br/> <span class="error">&#91;10&#93;</span> <a href="https://git.opendaylight.org/gerrit/#/c/56460/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/56460/</a><br/> <span class="error">&#91;11&#93;</span> <a href="https://lists.opendaylight.org/pipermail/release/2017-May/010653.html" class="external-link" target="_blank" rel="nofollow noopener">https://lists.opendaylight.org/pipermail/release/2017-May/010653.html</a></p> <p>This is still preventing Netconf features to included in Nitrogen Distribution.</p> <p>The most recent unsuccessful fix: <span class="error">&#91;12&#93;</span>.</p> <p><span class="error">&#91;12&#93;</span> <a href="https://git.opendaylight.org/gerrit/60669" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/60669</a></p> <p>I have just logs from failing distribution-check job on patch <a href="https://git.opendaylight.org/gerrit/#/c/59607/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/59607/</a>. Specifically the job <a href="https://jenkins.opendaylight.org/releng/job/distribution-check-nitrogen/250/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/job/distribution-check-nitrogen/250/</a>.</p> <p>I guess the problem could be that bundle config-persister-impl is being refreshed, that interrupts the netty-threadgroup-fixed's initialization and ends in "Server is unhealthy" error being emitted.</p> <p>2017-07-28 07:57:09,451 | INFO | pool-2-thread-1 | FeaturesServiceImpl | 6 - org.apache.karaf.features.core - 4.0.9 | Stopping bundles:<br/> .<br/> .<br/> 2017-07-28 07:57:09,454 | INFO | pool-2-thread-1 | FeaturesServiceImpl | 6 - org.apache.karaf.features.core - 4.0.9 | org.opendaylight.controller.config-persister-impl/0.7.0.SNAPSHOT</p> <p>After that we see the "Server is unhealthy" message:</p> <p>2017-07-28 07:57:09,466 | ERROR | config-pusher | ConfigRegistryImpl | 100 - org.opendaylight.controller.config-manager - 0.7.0.SNAPSHOT | Configuration Transaction failed on 2PC, server is unhealthy<br/> java.lang.IllegalStateException: Error - getInstance() failed for ModuleIdentifier</p> {factoryName='netty-threadgroup-fixed', instanceName='global-worker-group'} <p> in transaction TransactionIdentifier</p> {name='ConfigTransaction-5-7'} <p> at <br/> ...<br/> at org.opendaylight.controller.config.persist.impl.ConfigPusherImpl.pushConfig(ConfigPusherImpl.java:291)<span class="error">&#91;105:org.opendaylight.controller.config-persister-impl:0.7.0.SNAPSHOT&#93;</span><br/> at org.opendaylight.controller.config.persist.impl.ConfigPusherImpl.pushConfigWithConflictingVersionRetries(ConfigPusherImpl.java:159)<span class="error">&#91;105:org.opendaylight.controller.config-persister-impl:0.7.0.SNAPSHOT&#93;</span><br/> at org.opendaylight.controller.config.persist.impl.ConfigPusherImpl.internalPushConfigs(ConfigPusherImpl.java:124)<span class="error">&#91;105:org.opendaylight.controller.config-persister-impl:0.7.0.SNAPSHOT&#93;</span><br/> at org.opendaylight.controller.config.persist.impl.ConfigPusherImpl.processSingle(ConfigPusherImpl.java:78)<span class="error">&#91;105:org.opendaylight.controller.config-persister-impl:0.7.0.SNAPSHOT&#93;</span><br/> at org.opendaylight.controller.config.persist.impl.ConfigPusherImpl.process(ConfigPusherImpl.java:70)<span class="error">&#91;105:org.opendaylight.controller.config-persister-impl:0.7.0.SNAPSHOT&#93;</span><br/> at org.opendaylight.controller.config.persist.impl.osgi.ConfigPersisterActivator$2.run(ConfigPersisterActivator.java:131)<span class="error">&#91;105:org.opendaylight.controller.config-persister-impl:0.7.0.SNAPSHOT&#93;</span><br/> at java.lang.Thread.run(Thread.java:748)<span class="error">&#91;:1.8.0_131&#93;</span><br/> Caused by: org.opendaylight.controller.config.api.osgi.ServiceNotFoundException: Wait for OSGi service interface io.netty.channel.EventLoopGroup was interrrupted<br/> at org.opendaylight.controller.config.api.osgi.WaitingServiceTracker.waitForService(WaitingServiceTracker.java:54)<br/> at org.opendaylight.controller.config.yang.netty.threadgroup.NettyThreadgroupModule.createInstance(NettyThreadgroupModule.java:58)<br/> at org.opendaylight.controller.config.spi.AbstractModule.getInstance(AbstractModule.java:87)<br/> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)<span class="error">&#91;:1.8.0_131&#93;</span><br/> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)<span class="error">&#91;:1.8.0_131&#93;</span><br/> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)<span class="error">&#91;:1.8.0_131&#93;</span><br/> at java.lang.reflect.Method.invoke(Method.java:498)<span class="error">&#91;:1.8.0_131&#93;</span><br/> at org.opendaylight.controller.config.manager.impl.dependencyresolver.DependencyResolverManager$ModuleInvocationHandler.handleInvocation(DependencyResolverManager.java:148)<br/> at com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:84)<br/> at com.sun.proxy.$Proxy39.getInstance(Unknown Source)<br/> at org.opendaylight.controller.config.manager.impl.ConfigTransactionControllerImpl.secondPhaseCommit(ConfigTransactionControllerImpl.java:439)<br/> ... 35 more</p> <p>From the logs I cannot really tell what exactly caused refresh of config-persister-impl bundle. NETCONF project's features shouldn't depend on that kind of CSS stuff anymore.</p> <p>After testing the different netconf features, I figured out the one creating the problem is odl-netconf-ssh. Once I removed the prerequesite dependencies for this feature (not sure why this is required), I do not see the problem of server unhealthy anymore:</p> <p><a href="https://git.opendaylight.org/gerrit/#/c/60915/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/60915/</a></p> <p>&gt; <a href="https://git.opendaylight.org/gerrit/#/c/60915/1" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/60915/1</a></p> <p><a href="https://jenkins.opendaylight.org/releng/view/integration/job/integration-multipatch-test-nitrogen/20/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/integration/job/integration-multipatch-test-nitrogen/20/</a><br/> is building a zip to use in Sandbox.</p> <p>&gt; <a href="https://git.opendaylight.org/gerrit/#/c/60915/1" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/60915/1</a></p> <p>Merged. We can also confirm that "server is unhealthy" does not happen. Marking this as FIXED.</p> <p>Sandbox CSIT run <span class="error">&#91;17&#93;</span> ended with basically everything failing, but that is for other Bugs to track.</p> <p><span class="error">&#91;17&#93;</span> <a href="https://jenkins.opendaylight.org/sandbox/job/netconf-csit-1node-userfeatures-only-nitrogen/1/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/sandbox/job/netconf-csit-1node-userfeatures-only-nitrogen/1/</a></p> Blocks NETCONF-412 ODLPARENT-90 AAA-126 Duplicate NETCONF-414 Development External issue ID 7891 External issue URL Issue Type ODL SR Target Milestone Rank 0|i01y1r: