https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/CONTROLLER-1764 controller <p>The org.opendaylight.controller.md.sal.trace.dom.impl.TracingBroker JavaDoc says that one can add the mdsal-trace-features to the POM of a project's Karaf distribution (e.g. as done in <a href="https://git.opendaylight.org/gerrit/#/c/62403/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/62403/</a> for netvirt) and that one should be able to then just do, first:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>opendaylight-user@root&gt;feature:install odl-mdsal-trace followed by (after!) then any other regular feature, such as: opendaylight-user@root&gt;feature:install odl-netvirt-openstack</pre> </div></div> <p>and that this would enable the TracingDOMDataBroker. However, on a current master this does not work anymore, presumably following an impact of the Karaf 4 migration which causes wrong feature/bundle reloads which did not happen before, and the e.g. odl-netvirt-openstack installation fails with:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Exception in thread "config-blank-txn-0" java.lang.IllegalStateException: Error while copying old configuration from ModuleInternalInfo [name=ModuleIdentifier{factoryName='distributed-operational-datastore-provider', instanceName='distributed-operational-store-module'}] to org.opendaylight.controller.config.yang.config.distributed_datastore_provider.DistributedOperationalDataStoreProviderModuleFactory@9897f5a Caused by: java.lang.ClassCastException: org.opendaylight.controller.config.yang.config.distributed_datastore_provider.OperationalProperties cannot be cast to org.opendaylight.controller.config.yang.config.distributed_datastore_provider.OperationalProperties</pre> </div></div> <p>Resolving this is a pre-req. for <a href="https://jira.opendaylight.org/browse/CONTROLLER-1760" title="Tooling to find the real root cause culprit of memory leaks related to non-closed transactions (and tx chains)" class="issue-link" data-issue-key="CONTROLLER-1760"><del>CONTROLLER-1760</del></a> (which is req. for <a href="https://jira.opendaylight.org/browse/CONTROLLER-1756" title="OOM due to huge Map in ShardDataTree" class="issue-link" data-issue-key="CONTROLLER-1756"><del>CONTROLLER-1756</del></a>).</p> <p>Operating System: All<br/> Platform: All</p> CONTROLLER-1764

Karaf 4: odl-mdsal-trace cannot "just" be installed e.g. before odl-netvirt-openstack anymore, as it used to

Bug Resolved Done Stephen Kitt Michael Vorburger Tue, 29 Aug 2017 13:04:11 +0000 Tue, 27 Feb 2018 09:06:09 +0000 Tue, 27 Feb 2018 09:06:09 +0000 Nitrogen Oxygen Oxygen mdsal 0 5 <p>FTR: <a href="https://lists.opendaylight.org/pipermail/controller-dev/2017-August/013861.html" class="external-link" target="_blank" rel="nofollow noopener">https://lists.opendaylight.org/pipermail/controller-dev/2017-August/013861.html</a> is a thread with Tom on controller-dev that clarifies that we cannot solve the problem shown above simply by removing the (unused..) CSS from controller, just yet. (So I'll look more into how else this could be done.)</p> <p>Right... the trick (which I momentarily forgot) to figure out this kind of problem is to use <del>v on feature:install (along with -t to only simulate), using which we can see that we are "Refreshing bundles:" which -after a while of it seeming "stuck"</del> ultimately causes the error above - before it even does "Starting bundles:".</p> <p>AFAIK this is a problem that was fought with during the Karaf 4 migration ... how did we eventually overcome it there? What was the trick to avoid the refresh - how? </p> <p>Or is there no "trick", you have to figure out and fix the cause of the refresh? If I understand what it's trying to tell us below correctly, and trace back, the root cause is this one? Is that because we have both a javax.servlet-api/3.0.1 as well as a 3.1.0 (beware; confusing 3.0.1 VS 3.1.0) ? What I don't understand is that this is during feature:install odl-netvirt-openstack, but the initial feature:install odl-mdsal-trace adds no servlet-api, so why does this problem only happen if you install odl-netvirt-openstack after odl-mdsal-trace and not if you only install odl-netvirt-openstack?!</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>org.jboss.netty/3.10.6.Final (Should be wired to: javax.servlet-api/3.1.0 (through [org.jboss.netty/3.10.6.Final] osgi.wiring.package; filter:="(osgi.wiring.package=javax.servlet)"; resolution:=optional)) ? Refreshing bundles: com.typesafe.akka.actor/2.4.18 (Wired to org.opendaylight.controller.sal-distributed-datastore/1.7.0.SNAPSHOT which is being refreshed) com.typesafe.akka.cluster/2.4.18 (Wired to com.typesafe.akka.actor/2.4.18 which is being refreshed) com.typesafe.akka.osgi/2.4.18 (Wired to com.typesafe.akka.actor/2.4.18 which is being refreshed) com.typesafe.akka.persistence/2.4.18 (Wired to com.typesafe.akka.actor/2.4.18 which is being refreshed) com.typesafe.akka.remote/2.4.18 (Wired to org.jboss.netty/3.10.6.Final which is being refreshed) com.typesafe.akka.slf4j/2.4.18 (Wired to com.typesafe.akka.actor/2.4.18 which is being refreshed) com.typesafe.akka.stream/2.4.18 (Wired to com.typesafe.akka.actor/2.4.18 which is being refreshed) config-persister-feature-adapter/0.8.0.SNAPSHOT (Wired to org.opendaylight.controller.config-persister-impl/0.8.0.SNAPSHOT which is being refreshed) org.apache.karaf.config.core/4.0.9 (Should be wired to: org.apache.felix.metatype/1.1.2 (through [org.apache.karaf.config.core/4.0.9] osgi.wiring.package; filter:="(&amp;(osgi.wiring.package=org.osgi.service.metatype)(version&gt;=1.2.0)(!(version&gt;=2.0.0)))"; resolution:=optional)) org.apache.servicemix.bundles.jasypt/1.9.2.1 (Should be wired to: javax.servlet-api/3.1.0 (through [org.apache.servicemix.bundles.jasypt/1.9.2.1] osgi.wiring.package; filter:="(osgi.wiring.package=javax.servlet)"; resolution:=optional)) org.eclipse.persistence.core/2.6.2.v20151217-774c696 (Should be wired to: org.eclipse.persistence.antlr/3.2.0.v201302191141 (through [org.eclipse.persistence.core/2.6.2.v20151217-774c696] osgi.wiring.package; filter:="(osgi.wiring.package=org.eclipse.persistence.internal.libraries.antlr.runtime)"; resolution:=optional), javax.mail/1.4.4 (through [org.eclipse.persistence.core/2.6.2.v20151217-774c696] osgi.wiring.package; filter:="(osgi.wiring.package=javax.mail)"; resolution:=optional)) org.eclipse.persistence.moxy/2.6.2.v20151217-774c696 (Wired to org.eclipse.persistence.core/2.6.2.v20151217-774c696 which is being refreshed) org.jboss.netty/3.10.6.Final (Should be wired to: javax.servlet-api/3.1.0 (through [org.jboss.netty/3.10.6.Final] osgi.wiring.package; filter:="(osgi.wiring.package=javax.servlet)"; resolution:=optional)) org.opendaylight.controller.cds-access-api/1.3.0.SNAPSHOT (Wired to com.typesafe.akka.actor/2.4.18 which is being refreshed) org.opendaylight.controller.cds-access-client/1.3.0.SNAPSHOT (Wired to com.typesafe.akka.actor/2.4.18 which is being refreshed) org.opendaylight.controller.config-persister-directory-xml-adapter/0.8.0.SNAPSHOT (Wired to org.opendaylight.controller.config-persister-impl/0.8.0.SNAPSHOT which is being refreshed) org.opendaylight.controller.config-persister-file-xml-adapter/0.8.0.SNAPSHOT (Wired to org.opendaylight.controller.config-persister-impl/0.8.0.SNAPSHOT which is being refreshed) org.opendaylight.controller.config-persister-impl/0.8.0.SNAPSHOT (Wired to org.eclipse.persistence.moxy/2.6.2.v20151217-774c696 which is being refreshed) org.opendaylight.controller.sal-akka-raft/1.7.0.SNAPSHOT (Wired to com.typesafe.akka.actor/2.4.18 which is being refreshed) org.opendaylight.controller.sal-cluster-admin-impl/1.7.0.SNAPSHOT (Wired to com.typesafe.akka.actor/2.4.18 which is being refreshed) org.opendaylight.controller.sal-clustering-commons/1.7.0.SNAPSHOT (Wired to com.typesafe.akka.remote/2.4.18 which is being refreshed) org.opendaylight.controller.sal-distributed-datastore/1.7.0.SNAPSHOT (Wired to org.opendaylight.controller.sal-clustering-commons/1.7.0.SNAPSHOT which is being refreshed) org.opendaylight.controller.sal-remoterpc-connector/1.7.0.SNAPSHOT (Wired to org.opendaylight.controller.sal-clustering-commons/1.7.0.SNAPSHOT which is being refreshed) org.ops4j.pax.jdbc.config/1.0.1 (Wired to org.apache.servicemix.bundles.jasypt/1.9.2.1 which is being refreshed)</pre> </div></div> <p>&gt; because we have both a javax.servlet-api/3.0.1 as well as a 3.1.0</p> <p>not sure that's got anything to do with it; as said above in "why does this problem only happen if you install odl-netvirt-openstack after odl-mdsal-trace and not if you only install odl-netvirt-openstack" .. the odl-mdsal-trace feature actually does NOT install anything bringing along javax.servlet, so the analysis above may be completely wrong.</p> <p>Reducing issue Importance from major to minor, because (although it would be good to sort this out later, if only to learn more about Karaf features), there is actually an easy enough work-around, which is good enough for me for now for local testing, and unblocks me from proceeding with <a href="https://jira.opendaylight.org/browse/CONTROLLER-1760" title="Tooling to find the real root cause culprit of memory leaks related to non-closed transactions (and tx chains)" class="issue-link" data-issue-key="CONTROLLER-1760"><del>CONTROLLER-1760</del></a> related work:</p> <p>feature:install <span class="error">&#91;-v&#93;</span> odl-mdsal-trace</p> <p>feature:install <span class="error">&#91;-v&#93;</span> --no-auto-refresh odl-netvirt-openstack</p> <p>So the --no-auto-refresh (AKA -r) flag prevents the above problem.</p> <p>&gt; Reducing issue Importance from major to minor, because <br/> &gt; there is actually an easy enough work-around</p> <p>which is N/A when using featuresBoot, so if it could somehow sort this out after all, then (I'm told) it would make Jamo's and Sam's life for the CSIT integration idea in <a href="https://jira.opendaylight.org/browse/NETVIRT-878" title="CSIT should help to detect possible memory leaks leading to OOM related to non-closed transactions (and tx chains) early" class="issue-link" data-issue-key="NETVIRT-878">NETVIRT-878</a> quite a bit easier..</p> <p>Adding this under umbrella <a href="https://jira.opendaylight.org/browse/ODLPARENT-110" title="[Karaf 4] Avoid bundle refreshes" class="issue-link" data-issue-key="ODLPARENT-110"><del>ODLPARENT-110</del></a>. See comments there on how to possibly fix this on Odlparent side.</p> <p>Building odlparent with <a href="https://git.opendaylight.org/gerrit/65932" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/65932</a> followed by controller with <a href="https://git.opendaylight.org/gerrit/65933" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/65933</a> allows odl-mdsal-trace to be loaded, followed by odl-netvirt-openstack, without any breaking refreshes. (It still refreshes org.apache.karaf.config.core/4.0.10 which is unfortunate but doesn’t seem to cause issues.)</p> <p><a href="https://git.opendaylight.org/gerrit/#/c/65933/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/65933/</a> just merged.</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=jhershbe" class="user-hover" rel="jhershbe">jhershbe</a> do you want to re-test day after tomorrow or (using mvn -U) if this works for you now and we can close this as Fixed?</p> <p>Seeing <a href="https://jira.opendaylight.org/browse/NETVIRT-878" title="CSIT should help to detect possible memory leaks leading to OOM related to non-closed transactions (and tx chains) early" class="issue-link" data-issue-key="NETVIRT-878">NETVIRT-878</a> moving, it seems to me that this was fixed at least in Oxygen if not in Nitrogen - yes?</p> <p>I tried this in a recent Oxygen distro and it seems like it might be ok. There was some super ugly stuff</p> <p>coming from netconf and bierman02 though. Not sure the state of things after installing like this though.</p> Blocks NETVIRT-878 CONTROLLER-1760 ODLPARENT-110 ODLPARENT-132 Development External issue ID 9068 External issue URL Priority Rank 0|i02sl3: