[INTDIST-12] karaf distribution does not work offline Created: 17/Sep/14  Updated: 20/Oct/17  Resolved: 19/Nov/16

Status: Resolved
Project: integration-distribution
Component/s: Karaf
Affects Version/s: unspecified
Fix Version/s: None

Type: Bug
Reporter: Ed Warnicke Assignee: Vratko Polak
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Operating System: All
Platform: All


Issue Links:
Blocks
is blocked by CONTROLLER-961 Update Karaf to 3.0.2 Resolved
is blocked by CONTROLLER-1227 Karaf 3.0.3 upgrade Resolved
Duplicate
duplicates INTDIST-31 ODL does not start without an interne... Resolved
External issue ID: 1918
Priority: Low

 Description   

The karaf distribution does not work offline, it needs to download things from maven central to function.



 Comments   
Comment by Luis Gomez [ 22/Sep/14 ]

I just tested with latest RC1 candidate and looks fixed. I can start the compatible-with-all distro with no internet connection.

Comment by Vratko Polak [ 02/Oct/14 ]

In this Helium build:
http://nexus.opendaylight.org/content/repositories/automatedweeklyreleases-1030/org/opendaylight/integration/distribution-karaf/0.2.0-Helium/distribution-karaf-0.2.0-Helium.tar.gz
I have encountered
Error executing command: Error resolving artifact org.json:json:jar:20080701
Later version 20131018 is present in local repository, but that seem to be not enough for feature odl-mdsal-apidocs.
Reopening this bug.

Comment by Luis Gomez [ 02/Oct/14 ]

I confirm this observation, after starting karaf with no internet connection, I get this in the log:

2014-10-01 03:56:07,030 | WARN | Event Dispatcher | AetherBasedResolver | 5 - org.ops4j.pax.url.mvn - 1.6.0 | Error resolving artifactorg.apache.karaf.features:enterprise:xml:features:3.0.1:Could not transfer artifact org.apache.karaf.features:enterprise:xml:features:3.0.1 from/to central (http://repo1.maven.org/maven2/): Error transferring file: repo1.maven.org
2014-10-01 03:56:07,037 | WARN | Event Dispatcher | FeaturesServiceImpl | 24 - org.apache.karaf.features.core - 3.0.1 | Unable to add features repository mvn:org.apache.karaf.features/enterprise/3.0.1/xml/features at startup
2014-10-01 03:56:07,077 | WARN | Event Dispatcher | AetherBasedResolver | 5 - org.ops4j.pax.url.mvn - 1.6.0 | Error resolving artifactorg.apache.karaf.features:spring:xml:features:3.0.1:Could not transfer artifact org.apache.karaf.features:spring:xml:features:3.0.1 from/to central (http://repo1.maven.org/maven2/): Error transferring file: repo1.maven.org
2014-10-01 03:56:07,082 | WARN | Event Dispatcher | FeaturesServiceImpl | 24 - org.apache.karaf.features.core - 3.0.1 | Unable to add features repository mvn:org.apache.karaf.features/spring/3.0.1/xml/features at startup
2014-10-01 03:56:07,549 | WARN | Event Dispatcher | AetherBasedResolver | 5 - org.ops4j.pax.url.mvn - 1.6.0 | Error resolving artifactorg.eclipse.equinox:region:jar:1.0.0.v20110506:Could not transfer artifact org.eclipse.equinox:region:jar:1.0.0.v20110506 from/to central (http://repo1.maven.org/maven2/): Error transferring file: repo1.maven.org
2014-10-01 03:56:07,555 | WARN | Event Dispatcher | FeaturesServiceImpl | 24 - org.apache.karaf.features.core - 3.0.1 | Error when installing feature region: java.io.IOException: Error resolving artifact org.eclipse.equinox:region:jar:1.0.0.v20110506: Could not transfer artifact org.eclipse.equinox:region:jar:1.0.0.v20110506 from/to central (http://repo1.maven.org/maven2/): Error transferring file: repo1.maven.org

BR/Luis

Comment by Colin Dixon [ 02/Oct/14 ]

Ed Warnicke says that this is likely to be because somebody is using the 2008 version or org.json and really needs to upgrade to the version everyone else is using.

We should not automatically pull in the 2008 version.

Comment by Luis Gomez [ 07/Nov/14 ]

This is fixed in Helium first SR.

Comment by Vratko Polak [ 12/Nov/14 ]

> This is fixed in Helium first SR.

And not fixed on master:
https://jenkins.opendaylight.org/integration/view/Integration%20jobs/job/integration-master-project-centralized-integration/2719/artifact/distributions/extra/karaf/target/distribution-karaf-0.3.0-SNAPSHOT.tar.gz

Installation of odl-mdsal-clustering feature triggered this:

2014-11-12 09:43:30,909 | ERROR | pool-19-thread-2 | ShellUtil | 37 - org.apache.karaf.shell.console - 3.0.1 | Exception caught while executing command
java.io.IOException: Error resolving artifact org.json:json:jar:20080701: Could not transfer artifact org.json:json:jar:20080701 from/to central (http://repo1.maven.org/maven2/): Error transferring file: repo1.maven.org

Comment by Moiz Raja [ 12/Nov/14 ]

How was this fixed in Helium-1? I do not any specific commit in git which may fix this issue.

Comment by Vratko Polak [ 12/Nov/14 ]

> https://jenkins.opendaylight.org/integration/view/Integration%20jobs/job/integration-master-project-centralized-integration/2719/artifact/distributions/extra/karaf/target/distribution-karaf-0.3.0-SNAPSHOT.tar.gz

> org.json:json:jar:20080701
Other offending artifacts:
org.javassist:javassist:jar:3.18.2-GA
org.codehaus.jettison:jettison:jar:1.1

Offending features:
No feature named 'odl-aaa-authn-plugin' with version '0.0.0' available

Comment by Moiz Raja [ 12/Nov/14 ]

Vratko, Could you delete ~/.m2/repository and try running karaf and see if that works better for you?

Comment by Vratko Polak [ 12/Nov/14 ]

> Vratko, Could you delete ~/.m2/repository and try running karaf and see if
> that works better for you?

My install scripts were already removing .m2/repository
I think "polluted .m2" is bug with different number than this "offline" bug.

Comment by Luis Gomez [ 13/Nov/14 ]

Actually I am not sure which test I run the other day, the truth of the matter is the current karaf distribution (master and stable/helium) does not work offline in some scenarios (by enabling some features). This is normally due to projects not using the right external artifact version (the one shipped in karaf). We will be working on setting a CI test to detect this but the community has to come back with a way to avoid this situation like for example set all external dependencies in odlparent and make all projects depend on this one.

Comment by Vratko Polak [ 05/Dec/14 ]

Thank Luis for creating Jenkins job to detect this bug on ODL master.
https://jenkins.opendaylight.org/integration/view/All/job/integration-deploy-controller-offline/

>> org.json:json:jar:20080701

> Ed Warnicke says that this is likely to be because somebody is using the 2008 version or org.json and really needs to upgrade to the version everyone else is using.

Turns out "somebody" is the odlparent project itself.
Change submissions to bump the json version to 20131018:
odlparent:helium https://git.opendaylight.org/gerrit/#/c/13414/
odlparent:master https://git.opendaylight.org/gerrit/#/c/13415/

After merge, the Jenkis job can be enabled again, to see the next symptom.

Comment by Luis Gomez [ 05/Dec/14 ]

Thanks Vratko, I sent a mail a few weeks ago to Ed for this and found 2 projects: yangtools and controller.

mininet@mininet-vm:~/controller-test/distribution-karaf-0.2.1-Helium-SR1\> grep -R json ./system/org/opendaylight/ | grep 2008701

./system/org/opendaylight/controller/features-restconf/1.1.1-Helium-SR1/features-restconf-1.1.1-Helium-SR1-features.xml: <bundle>wrap:mvn:org.json/json/20080701</bundle>

./system/org/opendaylight/odlparent/odlparent/1.4.3-Helium-SR1/odlparent-1.4.3-Helium-SR1.pom: <org.json.version>20080701</org.json.version>

Comment by Luis Gomez [ 12/Jan/15 ]

It still fails for json:

Error executing command: Error resolving artifact org.json:json:jar:20080701: Could not find artifact org.json:json:jar:20080701 in defaultlocal (file:/opt/jenkins-integration/.m2/repository/)

This time the problem is is AAA project:

mininet@mininet-vm:~/test/distribution-karaf-0.3.0-SNAPSHOT\> grep -R json ./system/org/opendaylight/ | grep 20080701
./system/org/opendaylight/aaa/features-aaa/0.2.0-SNAPSHOT/features-aaa-0.2.0-SNAPSHOT.xml: <bundle>wrap:mvn:org.json/json/20080701</bundle>
./system/org/opendaylight/aaa/features-aaa/0.2.0-SNAPSHOT/features-aaa-0.2.0-SNAPSHOT-features.xml: <bundle>wrap:mvn:org.json/json/20080701</bundle>

BR/Luis

Comment by Luis Gomez [ 13/Jan/15 ]

OK, AAA is fixed now, next complain:

Error executing command: Error resolving artifact org.javassist:javassist:jar:3.18.2-GA: Could not find artifact org.javassist:javassist:jar:3.18.2-GA in defaultlocal (file:/opt/jenkins-integration/.m2/repository/)

Projects using javassist:

mininet@mininet-vm:~/controller-master/distribution-karaf-0.3.0-SNAPSHOT\> grep -R javassist.version
system/org/opendaylight/snmp4sdn/commons.snmp4sdn/0.2.0-SNAPSHOT/commons.snmp4sdn-0.2.0-SNAPSHOT.pom: <javassist.version>3.17.1-GA</javassist.version>
system/org/opendaylight/snmp4sdn/commons.snmp4sdn/0.2.0-SNAPSHOT/commons.snmp4sdn-0.2.0-SNAPSHOT.pom: <version>${javassist.version}</version>
system/org/opendaylight/controller/commons.opendaylight/1.5.0-SNAPSHOT/commons.opendaylight-1.5.0-SNAPSHOT.pom: <version>${javassist.version}</version>
system/org/opendaylight/yangtools/yangtools-parent/0.7.0-SNAPSHOT/yangtools-parent-0.7.0-SNAPSHOT.pom: <version>${javassist.version}</version>
system/org/opendaylight/odlparent/odlparent/1.5.0-SNAPSHOT/odlparent-1.5.0-SNAPSHOT.pom: <javassist.version>3.18.2-GA</javassist.version>
system/org/opendaylight/odlparent/odlparent/1.5.0-SNAPSHOT/odlparent-1.5.0-SNAPSHOT.pom: <version>${javassist.version}</version>
system/org/opendaylight/snbi/snbi/1.1.0-SNAPSHOT/snbi-1.1.0-SNAPSHOT.pom: <javassist.version>3.17.1-GA</javassist.version>
system/org/apache/karaf/karaf/3.0.1/karaf-3.0.1.pom: <javassist.version>3.9.0.GA</javassist.version>

Comment by Vratko Polak [ 28/Jan/15 ]

Helium SR2 is near so I did some testing,
and INTEGRAT-12 seem to be fixed on stable/helium
(for some features, and probably not fixed on master).

Official verification would be
https://jenkins.opendaylight.org/integration/view/Deploy%20Jobs/job/integration-deploy-controller-offline/
(if it were not currently disabled).

Comment by Luis Gomez [ 29/Jan/15 ]

Hi Vratko,

I just enabled offline jobs and:

1) stable/helium complains about a feature called transaction:

Error executing command: No feature named 'transaction' with version '0.0.0' available

After some investigagtion this comes from odl-adsal-clustering:

opendaylight-user@root>feature:info odl-adsal-clustering
Feature odl-adsal-clustering 0.5.3-SNAPSHOT
Description:
OpenDaylight :: AD-SAL :: Clustering
Feature has no configuration
Feature has no configuration files
Feature depends on:
transaction 0.0.0
odl-base-felix-dm 0.0.0
odl-base-eclipselink-persistence 0.0.0
odl-adsal-core 0.8.3-SNAPSHOT
Feature contains followed bundles:
mvn:org.opendaylight.controller/clustering.services/0.5.3-SNAPSHOT
mvn:org.opendaylight.controller/clustering.services-implementation/0.4.5-SNAPSHOT
Feature has no conditionals.

2) master complains about:

Error executing command: Error resolving artifact commons-codec:commons-codec:jar:1.8: Could not find artifact commons-codec:commons-codec:jar:1.8 in defaultlocal (file:/opt/jenkins-integration/.m2/repository/)

After some investigation, this artifact is not in system folder.

BR/Luis

Comment by Vratko Polak [ 05/Feb/15 ]

Hi Luis.

This time I have looked at master branch. First of all I have noticed,
that distribution-test contains different version of commons-codec (1.2)
than distribution-karaf (1.8). That is not cool.

Than I have downloaded https://jenkins.opendaylight.org/integration/view/Integration%20jobs/job/integration-master-project-centralized-integration/3045/artifact/distributions/karaf/target/distribution-karaf-0.3.0-SNAPSHOT.tar.gz
changed logging to TRACE (and changed maxFileSize),
launched with internet access,
and installed every feature found in integration/features/test:

feature:install odl-aaa-api odl-aaa-authn odl-aaa-authz odl-aaa-sssd-plugin odl-aaa-netconf-plugin odl-adsal-all odl-adsal-compatibility odl-bgpcep-all odl-dlux-core odl-lispflowmapping-all odl-mdsal-broker odl-neutron-northbound odl-neutron-implementation odl-netconf-connector-ssh odl-nsf-all odl-openflowjava-protocol odl-openflowplugin-flow-services odl-openflowplugin-flow-services-rest odl-openflowplugin-flow-services-ui odl-openflowplugin-nxm-extensions odl-ovsdb-all odl-packetcable-all odl-sdninterfaceapp-all odl-restconf odl-sfc-all odl-sfclisp odl-sfcofl2 odl-snbi-all odl-topoprocessing-all odl-ttp-all odl-snmp4sdn-all odl-tcpmd5-all odl-groupbasedpolicy-ofoverlay odl-l2switch-switch odl-l2switch-switch-rest odl-l2switch-switch-ui odl-plugin2oc odl-ovsdb-openstack odl-ovsdb-ovssfc odl-vtn-manager-all

Actually, disk on VM got full, but before that these download attempts were logged:
ProxySelector Request for http://repo1.maven.org/maven2/org/apache/karaf/features/enterprise/3.0.1/enterprise-3.0.1-features.xml
ProxySelector Request for http://repo1.maven.org/maven2/org/apache/karaf/features/spring/3.0.1/spring-3.0.1-features.xml
ProxySelector Request for http://repo1.maven.org/maven2/org/ops4j/pax/cdi/pax-cdi-features/0.7.0/pax-cdi-features-0.7.0-features.xml
ProxySelector Request for http://repo1.maven.org/maven2/org/eclipse/equinox/region/1.0.0.v20110506/region-1.0.0.v20110506.jar
ProxySelector Request for http://repository.springsource.com/maven/bundles/release/org/eclipse/equinox/region/1.0.0.v20110506/region-1.0.0.v20110506.jar
ProxySelector Request for http://repository.springsource.com/maven/bundles/external/org/eclipse/equinox/region/1.0.0.v20110506/region-1.0.0.v20110506.jar
ProxySelector Request for http://www.terracotta.org/kit/reflector?pageID=update.properties&tc-version=2.8.3&id=-1407865164&platform=amd64&uptime-secs=1&source=ehcache-core&jvm-version=1.7.0_67&kitID=ehcache.default&tc-product=ehcache-core+2.8.3&jvm-name=Java+HotSpot%28TM%29+64-Bit+Server+VM&os-name=Linux&patch=UNKNOWN
ProxySelector Request for http://repo1.maven.org/maven2/org/javassist/javassist/3.18.2-GA/javassist-3.18.2-GA.jar
ProxySelector Request for http://repo1.maven.org/maven2/org/apache/aries/transaction/org.apache.aries.transaction.blueprint/1.0.1/org.apache.aries.transaction.blueprint-1.0.1.jar
ProxySelector Request for http://repo1.maven.org/maven2/org/apache/aries/transaction/org.apache.aries.transaction.manager/1.1.0/org.apache.aries.transaction.manager-1.1.0.jar
ProxySelector Request for http://repo1.maven.org/maven2/org/apache/aries/spifly/org.apache.aries.spifly.dynamic.bundle/1.0.0/org.apache.aries.spifly.dynamic.bundle-1.0.0.jar
ProxySelector Request for http://repo1.maven.org/maven2/org/ow2/asm/asm-all/4.0/asm-all-4.0.jar
ProxySelector Request for http://repo1.maven.org/maven2/commons-net/commons-net/3.0.1/commons-net-3.0.1.jar
ProxySelector Request for http://repo1.maven.org/maven2/org/slf4j/slf4j-jdk14/1.7.2/slf4j-jdk14-1.7.2.jar
ProxySelector Request for http://repo1.maven.org/maven2/org/slf4j/slf4j-nop/1.7.2/slf4j-nop-1.7.2.jar
ProxySelector Request for http://repo1.maven.org/maven2/org/slf4j/slf4j-simple/1.7.2/slf4j-simple-1.7.2.jar
ProxySelector Request for http://repo1.maven.org/maven2/org/slf4j/slf4j-api/1.7.2/slf4j-api-1.7.2.jar
ProxySelector Request for http://repo1.maven.org/maven2/org/apache/felix/org.apache.felix.fileinstall/3.1.6/org.apache.felix.fileinstall-3.1.6.jar
ProxySelector Request for http://repo1.maven.org/maven2/org/eclipse/jetty/orbit/javax.servlet.jsp/2.2.0.v201112011158/javax.servlet.jsp-2.2.0.v201112011158.jar
ProxySelector Request for http://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-annotations/2.3.0/jackson-annotations-2.3.0.jar
ProxySelector Request for http://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-core/2.3.0/jackson-core-2.3.0.jar
ProxySelector Request for http://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-databind/2.3.0/jackson-databind-2.3.0.jar
ProxySelector Request for http://repo1.maven.org/maven2/org/codehaus/jettison/jettison/1.1/jettison-1.1.jar

That terracotta.org was accessed by thread _DEFAULT_ (no idea what that is),
others are maven resolutions.
As project odlparent is becoming the authority on third party versions,
maybe also repository locations can be put there, somehow?

Vratko.

Comment by Vratko Polak [ 12/Feb/15 ]

Looking at https://issues.apache.org/jira/browse/KARAF-2732
it seems we can do workarounds while using Karaf 3.0.1,
but 3.0.2 is different and ODL is going to upgrade anyway.
Bug tracking the upgrade is https://bugs.opendaylight.org/show_bug.cgi?id=2261
so I am setting this bug to depend on it.
Also, as there is no "Helium-3" milestone, changing target to Lithium M4.

Comment by Luis Gomez [ 27/Mar/15 ]

BTW here is the new job to track this:

https://jenkins.opendaylight.org/releng/view/integration/job/integration-distribution-offline-master/4/console

Apparently some features called transaction?? is currently failing the offline deployment.

After some investigagtion this comes from odl-adsal-clustering:

opendaylight-user@root>feature:info odl-adsal-clustering
Feature odl-adsal-clustering 0.5.3-SNAPSHOT
Description:
OpenDaylight :: AD-SAL :: Clustering
Feature has no configuration
Feature has no configuration files
Feature depends on:
transaction 0.0.0
odl-base-felix-dm 0.0.0
odl-base-eclipselink-persistence 0.0.0
odl-adsal-core 0.8.3-SNAPSHOT
Feature contains followed bundles:
mvn:org.opendaylight.controller/clustering.services/0.5.3-SNAPSHOT
mvn:org.opendaylight.controller/clustering.services-implementation/0.4.5-SNAPSHOT
Feature has no conditionals.

Is this something we can safely remove?

BR/Luis

Generated at Wed Feb 07 20:02:33 UTC 2024 using Jira 8.20.10#820010-sha1:ace47f9899e9ee25d7157d59aa17ab06aee30d3d.