https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/CONTROLLER-1499 controller <p>The following error seems to happen when config file is pushed, but only for features in featuresBoot, not when feature:install is issued at runtime.</p> <p>2016-03-16 04:48:42,313 | ERROR | config-pusher | ConfigPusherImpl | 108 - org.opendaylight.controller.config-persister-impl - 0.4.1.Beryllium-SR1 | Failed to apply configuration snapshot: 00-netty.xml(odl-mdsal-broker,odl-mdsal-broker)<br/> java.lang.SecurityException: Insufficient roles/credentials for operation<br/> at org.apache.karaf.management.KarafMBeanServerGuard.handleInvoke(KarafMBeanServerGuard.java:289)<span class="error">&#91;235:org.apache.karaf.management.server:3.0.3&#93;</span></p> <p>Also, this error does not happen on Beryllium snapshot builds. Attached TRACE logs when odl-config-netty is the feature present in featuresBoot.</p> <p>Operating System: All<br/> Platform: All</p> CONTROLLER-1499

SR1 candidate fails when booting with features

Bug Resolved Done Unassigned Vratko Polak Thu, 17 Mar 2016 09:34:55 +0000 Thu, 19 Oct 2017 22:33:48 +0000 Tue, 22 Mar 2016 10:07:52 +0000 Beryllium config 0 5 <p>Attachment karaf_log_sr1_2016317.tar.xz has been added with description: Archive with trace logs on SR1</p> <p>Also attached trace logs from Beryllium snapshot build. I did not spot any useful difference yet.</p> <p>Here is full error from SR1:<br/> 2016-03-17 08:52:53,936 | ERROR | config-pusher | ConfigPusherImpl | 98 - org.opendaylight.controller.config-persister-impl - 0.4.1.Beryllium-SR1 | Failed to apply configuration snapshot: 00-netty.xml(odl-config-netty,odl-config-netty)<br/> java.lang.SecurityException: Insufficient roles/credentials for operation<br/> at org.apache.karaf.management.KarafMBeanServerGuard.handleInvoke(KarafMBeanServerGuard.java:289)<span class="error">&#91;108:org.apache.karaf.management.server:3.0.3&#93;</span><br/> at org.apache.karaf.management.KarafMBeanServerGuard.invoke(KarafMBeanServerGuard.java:85)<span class="error">&#91;108:org.apache.karaf.management.server:3.0.3&#93;</span><br/> at org.apache.karaf.management.boot.KarafMBeanServerBuilder$MBeanInvocationHandler.invoke(KarafMBeanServerBuilder.java:63)<span class="error">&#91;karaf-jmx-boot.jar:&#93;</span><br/> at com.sun.proxy.$Proxy0.invoke(Unknown Source)<span class="error">&#91;:&#93;</span><br/> at com.sun.jmx.mbeanserver.MXBeanProxy$InvokeHandler.invoke(Unknown Source)<span class="error">&#91;:1.7.0_67&#93;</span><br/> at com.sun.jmx.mbeanserver.MXBeanProxy.invoke(Unknown Source)<span class="error">&#91;:1.7.0_67&#93;</span><br/> at javax.management.MBeanServerInvocationHandler.invoke(Unknown Source)<span class="error">&#91;:1.7.0_67&#93;</span><br/> at com.sun.proxy.$Proxy20.beginConfig(Unknown Source)<span class="error">&#91;66:org.opendaylight.controller.config-api:0.4.1.Beryllium-SR1&#93;</span><br/> at org.opendaylight.controller.config.util.ConfigRegistryJMXClient.beginConfig(ConfigRegistryJMXClient.java:104)<span class="error">&#91;90:org.opendaylight.controller.config-util:0.4.1.Beryllium-SR1&#93;</span><br/> at org.opendaylight.controller.config.facade.xml.transactions.TransactionProvider.getTestTransaction(TransactionProvider.java:117)<span class="error">&#91;96:org.opendaylight.controller.config-manager-facade-xml:0.4.1.Beryllium-SR1&#93;</span><br/> at org.opendaylight.controller.config.facade.xml.ConfigSubsystemFacade.test(ConfigSubsystemFacade.java:154)<span class="error">&#91;96:org.opendaylight.controller.config-manager-facade-xml:0.4.1.Beryllium-SR1&#93;</span><br/> at org.opendaylight.controller.config.facade.xml.ConfigSubsystemFacade.executeTests(ConfigSubsystemFacade.java:149)<span class="error">&#91;96:org.opendaylight.controller.config-manager-facade-xml:0.4.1.Beryllium-SR1&#93;</span><br/> at org.opendaylight.controller.config.facade.xml.ConfigSubsystemFacade.executeConfigExecution(ConfigSubsystemFacade.java:123)<span class="error">&#91;96:org.opendaylight.controller.config-manager-facade-xml:0.4.1.Beryllium-SR1&#93;</span><br/> at org.opendaylight.controller.config.persist.impl.ConfigPusherImpl.executeWithMissingModuleFactoryRetries(ConfigPusherImpl.java:329)<span class="error">&#91;98:org.opendaylight.controller.config-persister-impl:0.4.1.Beryllium-SR1&#93;</span><br/> at org.opendaylight.controller.config.persist.impl.ConfigPusherImpl.pushConfig(ConfigPusherImpl.java:305)<span class="error">&#91;98:org.opendaylight.controller.config-persister-impl:0.4.1.Beryllium-SR1&#93;</span><br/> at org.opendaylight.controller.config.persist.impl.ConfigPusherImpl.pushConfigWithConflictingVersionRetries(ConfigPusherImpl.java:164)<span class="error">&#91;98:org.opendaylight.controller.config-persister-impl:0.4.1.Beryllium-SR1&#93;</span><br/> at org.opendaylight.controller.config.persist.impl.ConfigPusherImpl.internalPushConfigs(ConfigPusherImpl.java:129)<span class="error">&#91;98:org.opendaylight.controller.config-persister-impl:0.4.1.Beryllium-SR1&#93;</span><br/> at org.opendaylight.controller.config.persist.impl.ConfigPusherImpl.processSingle(ConfigPusherImpl.java:83)<span class="error">&#91;98:org.opendaylight.controller.config-persister-impl:0.4.1.Beryllium-SR1&#93;</span><br/> at org.opendaylight.controller.config.persist.impl.ConfigPusherImpl.process(ConfigPusherImpl.java:74)<span class="error">&#91;98:org.opendaylight.controller.config-persister-impl:0.4.1.Beryllium-SR1&#93;</span><br/> at org.opendaylight.controller.config.persist.impl.osgi.ConfigPersisterActivator$2.run(ConfigPersisterActivator.java:131)<span class="error">&#91;98:org.opendaylight.controller.config-persister-impl:0.4.1.Beryllium-SR1&#93;</span><br/> at java.lang.Thread.run(Unknown Source)<span class="error">&#91;:1.7.0_67&#93;</span></p> <p>Attachment karaf_log_snapshot_20160317.tar.xz has been added with description: Archive with trace logs on Beryllium snapshot</p> <p>As Ryan Goulding points out, symptoms are similar to <a href="https://jira.opendaylight.org/browse/CONTROLLER-623" title="ManagementFactory.getPlatformMBeanServer() does not work in karaf" class="issue-link" data-issue-key="CONTROLLER-623"><del>CONTROLLER-623</del></a>.</p> <p>Still present on today's autorelease build.<br/> <a href="https://jenkins.opendaylight.org/releng/view/autorelease/job/aaa-csit-1node-authn-only-beryllium/1162/console" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/autorelease/job/aaa-csit-1node-authn-only-beryllium/1162/console</a></p> <p>This Bug is also present on Boron autorelease build. Not on Boron snapshot build, but last such distribution build is quite old (2016-03-13).</p> <p>I tried to reproduce on Beryllium snapshot codebase by using multipatch job <span class="error">&#91;0&#93;</span> on Sandbox. It built odlparent, yangtools, mdsal, controller, aaa and netconf, other parts of integration/distribution were downloaded from Nexus.<br/> Netconf userfeatures <del>only</del> job <span class="error">&#91;1&#93;</span> did not show this Bug, nor memory errors.</p> <p>It seems like root of this error is in autorelease.</p> <p>I have compared jars in system/ between SR1 and snapshot. All non-ODL artifacts were the same, all ODL artifacts were different (except org/opendaylight/controller/thirdparty/net.sf.jung2/2.0.1/net.sf.jung2-2.0.1.jar and org/opendaylight/controller/sal/0.7.0/sal-0.7.0.jar which I guess is correct).</p> <p>Perhaps the yang parser issue with not enough memory hits only on huge builds (explaining why docs build is faster when moved to front) and it can create (somehow) a broken build?</p> <p><span class="error">&#91;0&#93;</span> <a href="https://jenkins.opendaylight.org/sandbox/job/integration-multipatch-test-beryllium/1/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/sandbox/job/integration-multipatch-test-beryllium/1/</a><br/> <span class="error">&#91;1&#93;</span> <a href="https://jenkins.opendaylight.org/sandbox/job/netconf-csit-1node-userfeatures-only-beryllium/1/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/sandbox/job/netconf-csit-1node-userfeatures-only-beryllium/1/</a></p> <p>If I want to reproduce locally what are the steps to make this happen?</p> <p>(In reply to Thanh Ha from comment #6)<br/> &gt; If I want to reproduce locally what are the steps to make this happen?</p> <p>Figured it out, at minimum if you add to your etc/org.apache.karaf.features.cfg:</p> <p> featuresBoot=config,standard,region,package,kar,ssh,management,odl-integration-compatible-with-all,odl-restconf</p> <p>Should reproduce the issue.</p> <p>(In reply to Vratko Polák from comment #5)<br/> &gt; I tried to reproduce on Beryllium snapshot codebase by using multipatch job<br/> &gt; <span class="error">&#91;0&#93;</span> on Sandbox. It built odlparent, yangtools, mdsal, controller, aaa and<br/> &gt; netconf, other parts of integration/distribution were downloaded from Nexus.<br/> &gt; Netconf userfeatures <del>only</del> job <span class="error">&#91;1&#93;</span> did not show this Bug, nor memory errors.</p> <p>You might be on to something. I tried to do a minimal local build of autorelease building only odlparent, yangtools, mdsal, controller, aaa. I couldn't reproduce the problem either.</p> <p>Bizarre.</p> <p>Thanh, you can neglect odl-integration-compatible-with-all, odl-restconf is just enough to reproduce the issue.</p> <p>To test the theory that autorelease is too big I created a test job that splits the build into 3 parts. offset0, offset1, and offset2</p> <p><a href="https://jenkins.opendaylight.org/releng/view/autorelease/job/autorelease-release-test-beryllium/2/console" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/autorelease/job/autorelease-release-test-beryllium/2/console</a></p> <p>&gt; autorelease-release-test-beryllium/2/console</p> <p>That run did not ignore unit test failures, so we do not have resulting .zip.<br/> Also, perhaps offset 2 can be divided to smaller chunks?</p> <p>JJB's supposed to ignore manually created jobs. I guess there's a bug somewhere because builder removed the job <img class="emoticon" src="https://jira.opendaylight.org/images/icons/emoticons/sad.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>I'll submit a patch to build to add the test jobs.</p> <p>(In reply to Vratko Polák from comment #11)<br/> &gt; &gt; autorelease-release-test-beryllium/2/console<br/> &gt; <br/> &gt; That run did not ignore unit test failures, so we do not have resulting .zip.<br/> &gt; Also, perhaps offset 2 can be divided to smaller chunks?</p> <p>I can add another offset profile to split it more. Did you see it fail? I was asleep during the run so I wasn't able to watch it.</p> <p>Yes, it failed on a unit test failure. I am 80% sure it was the exact repeat of this one: <a href="https://jenkins.opendaylight.org/releng/view/autorelease/job/autorelease-release-beryllium/org.opendaylight.groupbasedpolicy$ofoverlay-renderer/90/testReport/junit/org.opendaylight.groupbasedpolicy.renderer.ofoverlay.arp/ArpSenderTest/floodArpTest/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/autorelease/job/autorelease-release-beryllium/org.opendaylight.groupbasedpolicy$ofoverlay-renderer/90/testReport/junit/org.opendaylight.groupbasedpolicy.renderer.ofoverlay.arp/ArpSenderTest/floodArpTest/</a></p> <p>Interesting my other attempt, build #96 appears to not have the test failures we saw in build #90</p> <p><a href="https://jenkins.opendaylight.org/releng/job/autorelease-release-beryllium/96/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/job/autorelease-release-beryllium/96/</a></p> <p>I did 2 changes for this run.</p> <p>1) Revert my 10g patch <a href="https://git.opendaylight.org/gerrit/36208" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/36208</a><br/> 2) Removed the redundant "install" goal <a href="https://git.opendaylight.org/gerrit/36420" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/36420</a></p> <p>The build is still running though so lets see if it fails with the same issues.</p> <p>This is looking very promising. So far aaa, capwap, etc have their tests passing. The fix seems to be working.</p> <p><a href="https://jenkins.opendaylight.org/releng/view/autorelease/job/integration-distribution-test-beryllium/346/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/autorelease/job/integration-distribution-test-beryllium/346/</a></p> <p>This does not happen on recent autorelease builds anymore, marking as fixed.</p> Development External issue ID 5546 External issue URL Issue Type ODL SR Target Milestone Rank 0|i02qy7: