[ODLPARENT-110] [Karaf 4] Avoid bundle refreshes Created: 02/Aug/17  Updated: 02/May/18  Resolved: 02/May/18

Status: Resolved
Project: odlparent
Component/s: General
Affects Version/s: 2.0.5
Fix Version/s: 2.0.6

Type: Bug
Reporter: Vratko Polak Assignee: Robert Varga
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Operating System: All
Platform: All

Issue Links:
Blocks
is blocked by CONTROLLER-1764 Karaf 4: odl-mdsal-trace cannot "just... Resolved
is blocked by YANGTOOLS-985 Install odl-restconf after odl-bgpcep... Resolved
External issue ID: 8923

 Description   

This is an umbrella item, feel free to open child items when it becomes clear which refreshes should be avoided and how.

Opening against Odlparent, as that project provided fix for CONTROLLER-1732.

The first issue is with adding Groupbasedpolicy features to Distribution. Unfortunately, I was not able to repeat the failure reliably. First, SingleFeatureTest frequently runs out of heap space, and the exact symptoms vary between runs.

I have used this [0] Odlparent patch to increase heap space for SFT JVM, and here is a segment related to refreshes as seen in surefire karaf log [1].

2017-08-02 11:20:01,180 | INFO | pool-2-thread-1 | FeaturesServiceImpl | 8 - org.apache.karaf.features.core - 4.0.9 | org.apache.karaf.shell.ssh/4.0.9
org.apache.karaf.shell.ssh/4.0.9
2017-08-02 11:20:01,181 | INFO | pool-2-thread-1 | CommandExtension | 45 - org.apache.karaf.shell.core - 4.0.9 | Unregistering commands for bundle org.apache.karaf.shell.ssh/4.0.9
2017-08-02 11:20:01,187 | INFO | pool-2-thread-1 | FeaturesServiceImpl | 8 - org.apache.karaf.features.core - 4.0.9 | org.ops4j.pax.logging.pax-logging-service/1.8.5
org.ops4j.pax.logging.pax-logging-service/1.8.5
Refreshing bundles:
io.netty.buffer/4.1.8.Final (Wired to io.netty.common/4.1.8.Final which is being refreshed)
io.netty.codec/4.1.8.Final (Wired to io.netty.buffer/4.1.8.Final which is being refreshed)
io.netty.codec-http/4.1.8.Final (Wired to io.netty.buffer/4.1.8.Final which is being refreshed)
io.netty.common/4.1.8.Final (Should be wired to: javassist/3.20.0.GA (through [io.netty.common/4.1.8.Final] osgi.wiring.package; filter:="(&(osgi.wiring.package=javassist)(version>=3.20.0)(!(version>=4.0.0)))"; resolution:=optional))
io.netty.handler/4.1.8.Final (Wired to io.netty.common/4.1.8.Final which is being refreshed)
io.netty.resolver/4.1.8.Final (Wired to io.netty.common/4.1.8.Final which is being refreshed)
io.netty.transport/4.1.8.Final (Wired to io.netty.resolver/4.1.8.Final which is being refreshed)
io.netty.transport-native-epoll/4.1.8.Final (Wired to io.netty.transport/4.1.8.Final which is being refreshed)
org.apache.karaf.shell.ssh/4.0.9 (Wired to org.apache.sshd.core/0.14.0 which is being refreshed)
org.apache.servicemix.bundles.jasypt/1.9.2.1 (Should be wired to: javax.servlet-api/3.1.0 (through [org.apache.servicemix.bundles.jasypt/1.9.2.1] osgi.wiring.package; filter:="(osgi.wiring.package=javax.servlet)"; resolution:=optional))
org.apache.sshd.core/0.14.0 (Should be wired to: bcprov/1.57.0 (through [org.apache.sshd.core/0.14.0] osgi.wiring.package; filter:="(&(osgi.wiring.package=org.bouncycastle.crypto.prng)(version>=1.51.0)(!(version>=2.0.0)))"; resolution:=optional), bcpkix/1.57.0 (through [org.apache.sshd.core/0.14.0] osgi.wiring.package; filter:="(&(osgi.wiring.package=org.bouncycastle.openssl)(version>=1.51.0)(!(version>=2.0.0)))"; resolution:=optional))
org.opendaylight.controller.netty-config-api/0.7.0.SNAPSHOT (Wired to io.netty.transport/4.1.8.Final which is being refreshed)
org.opendaylight.mdsal.model.ietf-lisp-address-types-2015-11-05/1.0.0.SNAPSHOT (Should be wired to: org.opendaylight.honeycomb.vbd.shared-models/1.6.0.SNAPSHOT (through [org.opendaylight.mdsal.model.ietf-lisp-address-types-2015-11-05/1.0.0.SNAPSHOT] osgi.wiring.package; filter:="(osgi.wiring.package=org.opendaylight.yang.gen.v1.urn.ietf.params.xml.ns.yang.ietf.lisp.address.types.rev151105)"))
org.opendaylight.netconf.api/1.3.0.SNAPSHOT (Wired to io.netty.transport/4.1.8.Final which is being refreshed)
org.opendaylight.netconf.mapping-api/1.3.0.SNAPSHOT (Wired to org.opendaylight.netconf.api/1.3.0.SNAPSHOT which is being refreshed)
org.opendaylight.netconf.protocol-framework/1.0.0.SNAPSHOT (Wired to io.netty.transport/4.1.8.Final which is being refreshed)
org.opendaylight.netconf.util/1.3.0.SNAPSHOT (Wired to io.netty.transport/4.1.8.Final which is being refreshed)
org.ops4j.pax.jdbc.config/1.0.1 (Wired to org.apache.servicemix.bundles.jasypt/1.9.2.1 which is being refreshed)
org.ops4j.pax.logging.pax-logging-service/1.8.5 (Should be wired to: javax.mail/1.4.4 (through [org.ops4j.pax.logging.pax-logging-service/1.8.5] osgi.wiring.package; filter:="(osgi.wiring.package=javax.mail)"; resolution:=optional))

Note that one refresh is still related to bcpkix even though that should have been avoided by [2].

By the way, how can I tell which commits on Odlparent 2.0.x branch are included in latest (2.0.0) release?

[0] https://git.opendaylight.org/gerrit/#/c/61034/1
[1] https://logs.opendaylight.org/sandbox/jenkins091/integration-multipatch-test-nitrogen/7/patch_tester/distribution/features/singles/odl-integration-compatible-with-all/target/surefire-reports/org.opendaylight.odlparent.featuretest.SingleFeatureTest-output.txt.gz
[2] https://git.opendaylight.org/gerrit/60545

 Comments   
Comment by Vratko Polak [ 03/Aug/17 ]

Another example, this time from Bier adding features to Distribution [3]:

Refreshing bundles:
io.netty.buffer/4.1.8.Final (Wired to io.netty.common/4.1.8.Final which is being refreshed)
io.netty.codec/4.1.8.Final (Wired to io.netty.buffer/4.1.8.Final which is being refreshed)
io.netty.codec-http/4.1.8.Final (Wired to io.netty.buffer/4.1.8.Final which is being refreshed)
io.netty.common/4.1.8.Final (Should be wired to: javassist/3.20.0.GA (through [io.netty.common/4.1.8.Final] osgi.wiring.package; filter:="(&(osgi.wiring.package=javassist)(version>=3.20.0)(!(version>=4.0.0)))"; resolution:=optional))
io.netty.handler/4.1.8.Final (Wired to io.netty.common/4.1.8.Final which is being refreshed)
io.netty.resolver/4.1.8.Final (Wired to io.netty.common/4.1.8.Final which is being refreshed)
io.netty.transport/4.1.8.Final (Wired to io.netty.resolver/4.1.8.Final which is being refreshed)
io.netty.transport-native-epoll/4.1.8.Final (Wired to io.netty.transport/4.1.8.Final which is being refreshed)
org.apache.karaf.shell.ssh/4.0.9 (Wired to org.apache.sshd.core/0.14.0 which is being refreshed)
org.apache.servicemix.bundles.jasypt/1.9.2.1 (Should be wired to: javax.servlet-api/3.1.0 (through [org.apache.servicemix.bundles.jasypt/1.9.2.1] osgi.wiring.package; filter:="(osgi.wiring.package=javax.servlet)"; resolution:=optional))
org.apache.sshd.core/0.14.0 (Should be wired to: bcpkix/1.57.0 (through [org.apache.sshd.core/0.14.0] osgi.wiring.package; filter:="(&(osgi.wiring.package=org.bouncycastle.openssl)(version>=1.51.0)(!(version>=2.0.0)))"; resolution:=optional), bcprov/1.57.0 (through [org.apache.sshd.core/0.14.0] osgi.wiring.package; filter:="(&(osgi.wiring.package=org.bouncycastle.crypto.prng)(version>=1.51.0)(!(version>=2.0.0)))"; resolution:=optional))
org.opendaylight.controller.netty-config-api/0.7.0.SNAPSHOT (Wired to io.netty.transport/4.1.8.Final which is being refreshed)
org.opendaylight.mdsal.model.ietf-lisp-address-types-2015-11-05/1.0.0.SNAPSHOT (Should be wired to: org.opendaylight.honeycomb.vbd.shared-models/1.6.0.SNAPSHOT (through [org.opendaylight.mdsal.model.ietf-lisp-address-types-2015-11-05/1.0.0.SNAPSHOT] osgi.wiring.package; filter:="(osgi.wiring.package=org.opendaylight.yang.gen.v1.urn.ietf.params.xml.ns.yang.ietf.lisp.address.types.rev151105)"))
org.opendaylight.netconf.api/1.3.0.SNAPSHOT (Wired to io.netty.transport/4.1.8.Final which is being refreshed)
org.opendaylight.netconf.mapping-api/1.3.0.SNAPSHOT (Wired to org.opendaylight.netconf.api/1.3.0.SNAPSHOT which is being refreshed)
org.opendaylight.netconf.protocol-framework/1.0.0.SNAPSHOT (Wired to io.netty.transport/4.1.8.Final which is being refreshed)
org.opendaylight.netconf.util/1.3.0.SNAPSHOT (Wired to io.netty.transport/4.1.8.Final which is being refreshed)
org.ops4j.pax.jdbc.config/1.0.1 (Wired to org.apache.servicemix.bundles.jasypt/1.9.2.1 which is being refreshed)
org.ops4j.pax.logging.pax-logging-service/1.8.5 (Should be wired to: javax.mail/1.4.4 (through [org.ops4j.pax.logging.pax-logging-service/1.8.5] osgi.wiring.package; filter:="(osgi.wiring.package=javax.mail)"; resolution:=optional))

[3] https://logs.opendaylight.org/releng/jenkins092/distribution-verify-nitrogen/318/features/singles/odl-integration-compatible-with-all/target/surefire-reports/org.opendaylight.odlparent.featuretest.SingleFeatureTest-output.txt.gz

Comment by Robert Varga [ 03/Aug/17 ]

There are a few problems here:

  • javax.mail being wired to pax-logging-service
  • javassist being wired to netty

also ssh being refreshed, but I am not sure why that is. The first two problems need a fix in odlparent.

Comment by Robert Varga [ 04/Aug/17 ]

The ssh refresh (and bc instsall) should be fixed by this: https://git.opendaylight.org/gerrit/61175

Netty reload will need and odlparent release and have it integrated into all projects using netty. Required odlparent patch is: https://git.opendaylight.org/gerrit/61174.

After odlparent is released, https://git.opendaylight.org/gerrit/61177 adjusts mdsal to not package javassist itself.

Comment by Robert Varga [ 04/Aug/17 ]

The ssh reload should be fixed. The netty reload should be fixed once odlparent-2.0.3 is released and adopted.

I am not sure it makes sense to also address pax-logging and jasypt – I think they are mostly harmless.

Comment by Vratko Polak [ 07/Aug/17 ]

This is the patch which needs this Bug fixed to pass verify: [7].

> The netty reload should be fixed once odlparent-2.0.3 is released and adopted.

I believe that is true. Sandbox run [8] confirms that, but it also included heap increase for SFT [9].

[7] https://git.opendaylight.org/gerrit/59755
[8] https://jenkins.opendaylight.org/sandbox/job/integration-multipatch-test-nitrogen/5
[9] https://git.opendaylight.org/gerrit/61034

Comment by Robert Varga [ 07/Aug/17 ]

Still needs more patches.

netconf:
https://git.opendaylight.org/gerrit/61273

Comment by Robert Varga [ 07/Aug/17 ]

aaa: https://git.opendaylight.org/gerrit/61275

Comment by Vratko Polak [ 07/Aug/17 ]

> Sandbox run [8] confirms that

I take that back, the job was (mis)configured to skip SFT.
Run #11 [11] shows bunch od bundles refreshing, possibly because the netconf patch has not been merged yet.

[11] https://jenkins.opendaylight.org/sandbox/job/integration-multipatch-test-nitrogen/11/

Comment by Vratko Polak [ 07/Aug/17 ]

> Run #11 [11] shows bunch od bundles refreshing

The only good news is that the current behavior does not depend on amount of heap space (1 GB or 4 GB) for SFT for the Groupbasedpolicy features.

Comment by Robert Varga [ 07/Aug/17 ]

odlparent patches:
2.0.x: https://git.opendaylight.org/gerrit/61285 BUG-8923: split out karaf-util
2.0.x: https://git.opendaylight.org/gerrit/61286 BUG-8923: use karaf-empty for SFT

Comment by Robert Varga [ 07/Aug/17 ]

Manual testing of the combination of netconf patch, the two odlparent patches and a rebuilt aaa seems to look okay.

The only refresh I am seeing now is:

efreshing bundles:
2017-08-07 18:57:54,959 | INFO | pool-2-thread-1 | FeaturesServiceImpl | 6 - org.apache.karaf.features.core - 4.0.9 | org.apache.karaf.config.core/4.0.9 (Should be wired to: org.apache.felix.metatype/1.1.2 (through [org.apache.karaf.config.core/4.0.9] osgi.wiring.package; filter:="(&(osgi.wiring.package=org.osgi.service.metatype)(version>=1.2.0)(!(version>=2.0.0)))"; resolution:=optional))
org.apache.karaf.config.core/4.0.9 (Should be wired to: org.apache.felix.metatype/1.1.2 (through [org.apache.karaf.config.core/4.0.9] osgi.wiring.package; filter:="(&(osgi.wiring.package=org.osgi.service.metatype)(version>=1.2.0)(!(version>=2.0.0)))"; resolution:=optional))
2017-08-07 18:57:54,959 | INFO | pool-2-thread-1 | FeaturesServiceImpl | 6 - org.apache.karaf.features.core - 4.0.9 | org.apache.servicemix.bundles.jasypt/1.9.2.1 (Should be wired to: javax.servlet-api/3.1.0 (through [org.apache.servicemix.bundles.jasypt/1.9.2.1] osgi.wiring.package; filter:="(osgi.wiring.package=javax.servlet)"; resolution:=optional))
org.apache.servicemix.bundles.jasypt/1.9.2.1 (Should be wired to: javax.servlet-api/3.1.0 (through [org.apache.servicemix.bundles.jasypt/1.9.2.1] osgi.wiring.package; filter:="(osgi.wiring.package=javax.servlet)"; resolution:=optional))
2017-08-07 18:57:54,959 | INFO | pool-2-thread-1 | FeaturesServiceImpl | 6 - org.apache.karaf.features.core - 4.0.9 | org.ops4j.pax.jdbc.config/1.0.1 (Wired to org.apache.servicemix.bundles.jasypt/1.9.2.1 which is being refreshed)
org.ops4j.pax.jdbc.config/1.0.1 (Wired to org.apache.servicemix.bundles.jasypt/1.9.2.1 which is being refreshed)

I think we should deal with these at some point, but certainly not before 2.0.3.

Comment by Robert Varga [ 07/Aug/17 ]

I forgot to note: mdsal was rebuilt, and the refresh report is from test output of all SFTs in netconf.

Comment by Vratko Polak [ 16/Aug/17 ]

I have encountered a new refresh while testing Bier features on Sandbox [14].

2017-08-16 10:49:43,982 | INFO | pool-2-thread-1 | FeaturesServiceImpl | 6 - org.apache.karaf.features.core - 4.0.9 | org.opendaylight.mdsal.model.ietf-lisp-address-types-2015-11-05/1.0.0.SNAPSHOT (Should be wired to: org.opendaylight.honeycomb.vbd.shared-models/1.6.0.SNAPSHOT (through [org.opendaylight.mdsal.model.ietf-lisp-address-types-2015-11-05/1.0.0.SNAPSHOT] osgi.wiring.package; filter:="(osgi.wiring.package=org.opendaylight.yang.gen.v1.urn.ietf.params.xml.ns.yang.ietf.lisp.address.types.rev151105)"))
org.opendaylight.mdsal.model.ietf-lisp-address-types-2015-11-05/1.0.0.SNAPSHOT (Should be wired to: org.opendaylight.honeycomb.vbd.shared-models/1.6.0.SNAPSHOT (through [org.opendaylight.mdsal.model.ietf-lisp-address-types-2015-11-05/1.0.0.SNAPSHOT] osgi.wiring.package; filter:="(osgi.wiring.package=org.opendaylight.yang.gen.v1.urn.ietf.params.xml.ns.yang.ietf.lisp.address.types.rev151105)"))

A believe that is a fault of Honeycomb/Vbd project, no action from Odlparent is needed.

[14] https://logs.opendaylight.org/sandbox/jenkins091/integration-multipatch-test-nitrogen/11/patch_tester/distribution/features/singles/odl-integration-compatible-with-all/target/surefire-reports/org.opendaylight.odlparent.featuretest.SingleFeatureTest-output.txt.gz

Comment by Vratko Polak [ 16/Aug/17 ]

> fault of Honeycomb/Vbd project

Opened https://bugs.opendaylight.org/show_bug.cgi?id=8997 to track that.

Generated at Wed Feb 07 20:27:39 UTC 2024 using Jira 8.20.10#820010-sha1:ace47f9899e9ee25d7157d59aa17ab06aee30d3d.