karaf.log
|
Relates |
|||
| relates to |
|
List of examples where separately rel... | Closed |
[ODLPARENT-144] SingleFeatureTest failures due to ScrServiceMBean Created: 14/Mar/18 Updated: 15/Mar/19 Resolved: 04/Jul/18 |
|
| Status: | Resolved |
| Project: | odlparent |
| Component/s: | SFT |
| Affects Version/s: | None |
| Fix Version/s: | 3.1.2, 4.0.0 |
| Type: | Bug | Priority: | High |
| Reporter: | Michael Vorburger | Assignee: | Stephen Kitt |
| Resolution: | Done | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Attachments: |
karaf.log
|
||||||||
| Issue Links: |
|
||||||||
| Description |
|
We're suddenly regularly seeing this SFT failure: Running org.opendaylight.odlparent.featuretest.SingleFeatureTest
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 332.267 sec <<< FAILURE! - in org.opendaylight.odlparent.featuretest.SingleFeatureTest
installFeatureCatchAndLog(org.opendaylight.odlparent.featuretest.SingleFeatureTest)[repoUrl: file:/w/workspace/neutron-maven-verify-oxygen-mvn33-openjdk8/features/production/odl-neutron-transcriber/target/feature/feature.xml, Feature: odl-neutron-transcriber 0.10.0.SNAPSHOT] Time elapsed: 331.59 sec <<< ERROR!
org.awaitility.core.ConditionTimeoutException: Condition with alias 'checkBundleDiagInfos' didn't complete within 300 seconds because lambda expression in org.opendaylight.odlparent.bundlestest.lib.TestBundleDiag: expected system either ready with all bundles Active, or Stopping or Failure (but not still booting in GracePeriod, Waiting, Starting, Unknown;but just Resolved and some exceptional Installed OK) but was <diag: Booting {Installed=0, Resolved=6, Unknown=0, GracePeriod=0, Waiting=1, Starting=0, Active=351, Stopping=0, Failure=0}
1. NOK org.apache.karaf.scr.management:4.1.3: OSGi state = Active, Karaf bundleState = Waiting, due to: Declarative Services
ScrServiceMBean (2)
missing references: mBeanServer
>.
at org.awaitility.core.ConditionAwaiter.await(ConditionAwaiter.java:136)
at org.awaitility.core.AbstractHamcrestCondition.await(AbstractHamcrestCondition.java:89)
at org.awaitility.core.ConditionFactory.until(ConditionFactory.java:840)
at org.awaitility.core.ConditionFactory.until(ConditionFactory.java:619)
at org.opendaylight.odlparent.bundlestest.lib.TestBundleDiag.checkBundleDiagInfos(TestBundleDiag.java:71)
at org.opendaylight.odlparent.bundlestest.lib.TestBundleDiag.checkBundleDiagInfos(TestBundleDiag.java:54)
at org.opendaylight.odlparent.featuretest.SingleFeatureTest.installFeature(SingleFeatureTest.java:404)
at org.opendaylight.odlparent.featuretest.SingleFeatureTest.installFeatureCatchAndLog(SingleFeatureTest.java:355)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang.Thread.run(Thread.java:748)
Results :
Tests in error:
Condition with alias 'checkBundleDiagInfos' didn't complete within 300 seconds because lambda expression in org.opendaylight.odlparent.bundlestest.lib.TestBundleDiag: expected system either ready with all bundles Active, or Stopping or Failure (but not still booting in GracePeriod, Waiting, Starting, Unknown;but just Resolved and some exceptional Installed OK) but was <diag: Booting {Installed=0, Resolved=6, Unknown=0, GracePeriod=0, Waiting=1, Starting=0, Active=351, Stopping=0, Failure=0}
1. NOK org.apache.karaf.scr.management:4.1.3: OSGi state = Active, Karaf bundleState = Waiting, due to: Declarative Services
ScrServiceMBean (2)
missing references: mBeanServer
e.g. here https://jenkins.opendaylight.org/releng/job/neutron-maven-verify-oxygen-mvn33-openjdk8/165/console (for https://git.opendaylight.org/gerrit/#/c/69397/, but totally unrelated) and here https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/autorelease-release-oxygen/215/console-timestamp.log.gz We got two in the past week, and at least I don't recall have seen this one before - have others? They are intermittent and disappear after "recheck" - but ideally should never happen. Should we just increase the SFT time-out? Should we whitelist the "org.apache.karaf.scr.management" bundle in SFT, as in "we REALLY don't care if some Karaf Service Component Runtime (SCR) OSGi Declarative Services (DS) stupid JMX monitoring bean failed to come up for some reason, as long as all of our other ODL features installed correctly" ? The odlparent project would certainly welcome anyone's contributions for this. |
| Comments |
| Comment by Michael Vorburger [ 15/Mar/18 ] |
|
> Should we just increase the SFT time-out? thought more about this, and realized that in theory we could perhaps try that first, and without changing odlparent code (and releasing, and bumping... OMG) by just setting -Dsft.diag.timeout=... BUT the current (default) value is already 300s (see message above) = 5 minutes, so I doubt increasing that would even solve this problem, I suspect it's more like some ... thing/problem in Karaf where occassionally that scr.management bundle just really fails to come up (and waiting longer would not help). Therefore whitelisting instead of increasing the time out is probably the right approach here. I could also be interesting to see if in the full Karaf LOG of this SFT failure there are details about why scr.management failed to start, but AFAIK we loose that in SFT (unless we -Dkaraf.keep.unpack=true). |
| Comment by Daniel Farrell [ 15/Mar/18 ] |
|
This has broken most of our recent autorelease-oxygen builds and is the main thing keeping us from getting Oxygen released. Is there really nothing we can do about it in the short term? |
| Comment by Michael Vorburger [ 15/Mar/18 ] |
|
dfarrell07 I'm puzzled how this "has broken" - neither Karaf nor SFT changed recently, actually... curious. |
| Comment by Michael Vorburger [ 15/Mar/18 ] |
|
https://git.opendaylight.org/gerrit/#/c/69525/ implements the org.apache.karaf.scr.management white listing. |
| Comment by Daniel Farrell [ 15/Mar/18 ] |
|
vorburger - Awesome, thanks! skitt is working on a patch to archive additional logs so we might be able to figure out the root cause. I have a discussion thread about doing that on integration-dev. |
| Comment by Jamo Luhrsen [ 15/Mar/18 ] |
|
is this another instance of this failure: https://jenkins.opendaylight.org/releng/view/autorelease/job/autorelease-release-oxygen/218/console also, the patch to get more logs was merged: |
| Comment by Stephen Kitt [ 19/Mar/18 ] |
|
So we got the awesome logs from the build... “Archiving archives.log” I guess that means we need some other releng-builder fixes to actually store the SFT logs! I’m releasing odlparent 3.0.3 anyway with Michael’s fix so that we can bump to that if necessary. (I see build 232 was good, hopefully we won’t need to...) |
| Comment by Sam Hague [ 15/May/18 ] |
|
Seen again in https://jenkins.opendaylight.org/releng/job/autorelease-release-fluorine/85/ |
| Comment by Sam Hague [ 29/May/18 ] |
|
Another one: https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/autorelease-release-oxygen/298 |
| Comment by Robert Varga [ 08/Jun/18 ] |
|
https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/autorelease-release-fluorine/109 is another one. The curios thing is, 2 out of three failures seen are in odl-netvirt-coe and the other one is in odl-netvirt-openstack, which runs just after it:
[INFO] OpenDaylight :: NetVirt :: impl 0.7.0 .............. SUCCESS [01:26 min] [INFO] ODL :: netvirt :: coe-impl 0.7.0 ................... SUCCESS [ 13.485 s] [INFO] OpenDaylight :: NetVirt :: Coe 0.7.0 ............... FAILURE [06:07 min] [INFO] OpenDaylight :: NetVirt :: OpenStack 0.7.0 ......... SKIPPED and: [INFO] OpenDaylight :: NetVirt :: impl 0.6.2 .............. SUCCESS [01:03 min] [INFO] ODL :: netvirt :: coe-impl 0.6.2 ................... SUCCESS [ 10.357 s] [INFO] OpenDaylight :: NetVirt :: Coe 0.6.2 ............... SUCCESS [01:11 min] [INFO] OpenDaylight :: NetVirt :: OpenStack 0.6.2 ......... FAILURE [05:58 min] [INFO] ODL :: sfc :: odl-sfc-genius 0.7.2 ................. SKIPPED is this a coincidence? |
| Comment by Robert Varga [ 08/Jun/18 ] |
|
The thing that disturbs me the most is in https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/autorelease-release-fluorine/109/netvirt/features/odl-netvirt-coe/target/surefire-reports/org.opendaylight.odlparent.featuretest.SingleFeatureTest-output.txt.gz : Stopping bundles: org.ops4j.pax.jdbc.config/1.1.0 org.apache.servicemix.bundles.jasypt/1.9.2.1 org.ops4j.pax.exam.inject/4.11.0 org.apache.geronimo.specs.geronimo-atinject_1.0_spec/1.0.0 org.ops4j.pax.logging.pax-logging-log4j2/1.10.1 Uninstalling bundles: org.apache.geronimo.specs.geronimo-atinject_1.0_spec/1.0.0 which is causing this: Refreshing bundles:
org.apache.geronimo.specs.geronimo-atinject_1.0_spec/1.0.0 (Bundle will be uninstalled)
org.apache.servicemix.bundles.jasypt/1.9.2.1 (Should be wired to: javax.servlet-api/3.1.0 (through [org.apache.servicemix.bundles.jasypt/1.9.2.1] osgi.wiring.package; filter:="(osgi.wiring.package=javax.servlet)"; resolution:=optional))
org.ops4j.pax.exam.inject/4.11.0 (Wired to org.apache.geronimo.specs.geronimo-atinject_1.0_spec/1.0.0 which is being refreshed)
org.ops4j.pax.jdbc.config/1.1.0 (Wired to org.apache.servicemix.bundles.jasypt/1.9.2.1 which is being refreshed)
org.ops4j.pax.logging.pax-logging-log4j2/1.10.1 (Should be wired to: com.fasterxml.jackson.core.jackson-annotations/2.8.0 (through [org.ops4j.pax.logging.pax-logging-log4j2/1.10.1] osgi.wiring.package; filter:="(osgi.wiring.package=com.fasterxml.jackson.annotation)"; resolution:=optional), com.fasterxml.jackson.core.jackson-databind/2.8.9 (through [org.ops4j.pax.logging.pax-logging-log4j2/1.10.1] osgi.wiring.package; filter:="(osgi.wiring.package=com.fasterxml.jackson.databind)"; resolution:=optional), com.fasterxml.jackson.core.jackson-core/2.8.9 (through [org.ops4j.pax.logging.pax-logging-log4j2/1.10.1] osgi.wiring.package; filter:="(osgi.wiring.package=com.fasterxml.jackson.core)"; resolution:=optional), stax2-api/3.1.4 (through [org.ops4j.pax.logging.pax-logging-log4j2/1.10.1] osgi.wiring.package; filter:="(osgi.wiring.package=org.codehaus.stax2)"; resolution:=optional), javax.mail/1.4.4 (through [org.ops4j.pax.logging.pax-logging-log4j2/1.10.1] osgi.wiring.package; filter:="(osgi.wiring.package=javax.mail)"; resolution:=optional), com.lmax.disruptor/3.3.10 (through [org.ops4j.pax.logging.pax-logging-log4j2/1.10.1] osgi.wiring.package; filter:="(osgi.wiring.package=com.lmax.disruptor)"; resolution:=optional), com.fasterxml.jackson.dataformat.jackson-dataformat-xml/2.8.9 (through [org.ops4j.pax.logging.pax-logging-log4j2/1.10.1] osgi.wiring.package; filter:="(osgi.wiring.package=com.fasterxml.jackson.dataformat.xml)"; resolution:=optional))
Now ... if we are re-wiring pax.exam, I suspect the test running on it is going to be ... less than stable. |
| Comment by Sam Hague [ 08/Jun/18 ] |
|
rovarga are you saying that during the SFT something is causing a refresh? Specific to netvirt-coe? We have seen this same issue in place like neutron and sfc and others also. |
| Comment by Sam Hague [ 08/Jun/18 ] |
|
seen in sfc job: Console Logs: Jenkins Build: |
| Comment by Robert Varga [ 11/Jun/18 ] |
|
Maybe, I am just gathering data, trying to formulate a theory about what is going on. |
| Comment by Robert Varga [ 11/Jun/18 ] |
|
After looking through the logs, the SFC case does not seem to indicate any particular failure. This does feel like an upstream bug The reason why SCR is being pulled in is ... drumroll ... pax-http-jetty. Since management feature is always present (implied by features.boot), we also install the corresponding management component. Full wiring of the failied SFC feature:
opendaylight-user@root>feature:install --features-wiring odl-sfc-scf-openflow Wiring: bundle / 4.1.5 jaas-boot / 0.0.0 management / 4.1.5 jaas / 4.1.5 aries-blueprint / 4.1.5 aries-proxy / 4.1.5 pax-jdbc-config / 1.1.0 pax-jdbc-spec / 1.1.0 jaas / 4.1.5 jaas-boot / 0.0.0 pax-jdbc / 1.1.0 pax-jdbc-spec / 1.1.0 shell / 4.1.5 jaas-boot / 0.0.0 aries-blueprint-condition-bundle / 4.1.5 bundle / 4.1.5 aries-blueprint / 4.1.5 deployer-condition-wrap / 4.1.5 deployer / 4.1.5 wrap / 0.0.0 jaas-condition-aries_blueprint / 4.1.5 aries-blueprint / 4.1.5 jaas / 4.1.5 jaas-condition-shell / 4.1.5 shell / 4.1.5 jaas / 4.1.5 system-condition-shell / 4.1.5 shell / 4.1.5 system / 4.1.5 config-condition-shell / 4.1.5 shell / 4.1.5 config / 4.1.5 shell-compat / 4.1.5 shell / 4.1.5 aries-blueprint / 4.1.5 config-condition-management / 4.1.5 config / 4.1.5 management / 4.1.5 deployer-condition-req_osgi_extender_filter______osgi_extender_osgi_blueprint__version__1_0___ / 4.1.5 deployer / 4.1.5 aries-blueprint / 4.1.5 deployer-condition-kar / 4.1.5 kar / 4.1.5 deployer / 4.1.5 ssh / 4.1.5 shell / 4.1.5 jaas / 4.1.5 bundle-condition-management / 4.1.5 bundle / 4.1.5 management / 4.1.5 feature-condition-shell / 4.1.5 shell / 4.1.5 feature / 4.1.5 management-condition-aries_blueprint / 4.1.5 aries-blueprint / 4.1.5 management / 4.1.5 kar-condition-shell / 4.1.5 kar / 4.1.5 shell / 4.1.5 bundle-condition-shell / 4.1.5 shell / 4.1.5 bundle / 4.1.5 51b3d2b9-436d-4263-80a8-dbb1cfb39b85 / 0.0.0 framework / 4.1.5 standard / 4.1.5 ssh / 4.1.5 standard / 4.1.5 kar / 4.1.5 diagnostic / 4.1.5 log / 4.1.5 instance / 4.1.5 bundle / 4.1.5 aries-blueprint / 4.1.5 jaas / 4.1.5 management / 4.1.5 service / 4.1.5 system / 4.1.5 package / 4.1.5 shell-compat / 4.1.5 shell / 4.1.5 deployer / 4.1.5 feature / 4.1.5 ssh / 4.1.5 config / 4.1.5 wrap / 0.0.0 Wiring: odl-akka-leveldb-0.7 / 3.1.0 wrap / 0.0.0 odl-yangtools-export / 2.0.5 odl-yangtools-parser-api / 2.0.5 odl-guava-23 / 3.1.1 wrap / 0.0.0 odl-config-manager-facade-xml / 0.9.0.SNAPSHOT odl-config-core / 0.9.0.SNAPSHOT odl-ovsdb-hwvtepsouthbound-api / 1.7.0.SNAPSHOT odl-mdsal-models / 0.13.0.SNAPSHOT odl-config-api / 0.9.0.SNAPSHOT odl-mdsal-binding-base / 2.5.0.SNAPSHOT pax-http / 6.0.9 pax-jetty / 9.3.21.v20170918 odl-karaf-feat-jetty / 3.1.0 war / 4.1.5 odl-openflowplugin-libraries / 0.7.0.SNAPSHOT odl-guava-23 / 3.1.1 pax-http-whiteboard / 6.0.9 pax-http / 6.0.9 odl-karaf-feat-jdbc / 3.1.0 jdbc / 4.1.5 aries-blueprint / 4.1.5 aries-proxy / 4.1.5 bundle / 4.1.5 jaas-boot / 0.0.0 odl-infrautils-inject / 1.4.0.SNAPSHOT wrap / 0.0.0 management / 4.1.5 jaas / 4.1.5 pax-jdbc / 1.1.0 pax-jdbc-spec / 1.1.0 odl-openflowplugin-app-reconciliation-framework / 0.7.0.SNAPSHOT odl-openflowplugin-southbound / 0.7.0.SNAPSHOT odl-akka-clustering-2.5 / 3.1.0 odl-akka-system-2.5 / 3.1.0 odl-mdsal-eos-dom / 2.5.0.SNAPSHOT odl-mdsal-eos-common / 2.5.0.SNAPSHOT pax-http-service / 0.0.0 pax-http / 6.0.9 odl-triemap / 1.0.5 odl-guava-23 / 3.1.1 pax-jdbc-config / 1.1.0 pax-jdbc-spec / 1.1.0 odl-yangtools-codec / 2.0.5 odl-yangtools-data / 2.0.5 http / 4.1.5 pax-http-service / 0.0.0 pax-war / 6.0.9 pax-http-whiteboard / 6.0.9 jaas / 4.1.5 jaas-boot / 0.0.0 odl-mdsal-dom / 2.5.0.SNAPSHOT odl-mdsal-dom-broker / 2.5.0.SNAPSHOT odl-netty-4 / 3.1.0 odl-javassist-3 / 3.1.0 odl-aaa-api / 0.8.0.SNAPSHOT odl-mdsal-binding-base / 2.5.0.SNAPSHOT odl-akka-system-2.5 / 3.1.0 odl-akka-scala-2.12 / 3.1.0 shell / 4.1.5 jaas-boot / 0.0.0 odl-openflowplugin-nxm-extensions / 0.7.0.SNAPSHOT odl-openflowplugin-southbound / 0.7.0.SNAPSHOT odl-config-startup / 0.9.0.SNAPSHOT odl-config-persister / 0.9.0.SNAPSHOT scr-condition-bundle / 4.1.5 scr / 4.1.5 bundle / 4.1.5 odl-mdsal-eos-common / 2.5.0.SNAPSHOT wrap / 0.0.0 odl-yangtools-data-api / 2.0.5 odl-jolokia / 1.11.0.SNAPSHOT wrap / 0.0.0 http / 4.1.5 feature-condition-shell / 4.1.5 feature / 4.1.5 shell / 4.1.5 odl-mdsal-common / 2.5.0.SNAPSHOT wrap / 0.0.0 odl-yangtools-common / 2.0.5 odl-mdsal-models / 0.13.0.SNAPSHOT odl-mdsal-binding-base / 2.5.0.SNAPSHOT wrap / 0.0.0 deployer-condition-wrap / 4.1.5 wrap / 0.0.0 deployer / 4.1.5 odl-config-manager / 0.9.0.SNAPSHOT wrap / 0.0.0 odl-config-core / 0.9.0.SNAPSHOT odl-akka-persistence-2.5 / 3.1.0 odl-akka-system-2.5 / 3.1.0 odl-akka-leveldb-0.7 / 3.1.0 deployer-condition-kar / 4.1.5 deployer / 4.1.5 kar / 4.1.5 odl-yangtools-data-api / 2.0.5 wrap / 0.0.0 odl-yangtools-parser-api / 2.0.5 jaas-condition-shell / 4.1.5 jaas / 4.1.5 shell / 4.1.5 config-condition-management / 4.1.5 management / 4.1.5 config / 4.1.5 odl-mdsal-singleton-common / 2.5.0.SNAPSHOT wrap / 0.0.0 odl-mdsal-eos-common / 2.5.0.SNAPSHOT odl-yangtools-common / 2.0.5 wrap / 0.0.0 odl-yangtools-util / 2.0.5 odl-openflowplugin-app-forwardingrules-manager / 0.7.0.SNAPSHOT wrap / 0.0.0 odl-openflowplugin-southbound / 0.7.0.SNAPSHOT pax-http-jetty / 6.0.9 pax-jetty / 9.3.21.v20170918 scr / 4.1.5 odl-yangtools-parser / 2.0.5 wrap / 0.0.0 odl-yangtools-parser-api / 2.0.5 deployer-condition-req_osgi_extender_filter______osgi_extender_osgi_blueprint__version__1_0___ / 4.1.5 deployer / 4.1.5 aries-blueprint / 4.1.5 odl-restconf / 1.8.0.SNAPSHOT odl-jackson-2.8 / 3.1.0 odl-restconf-base / 1.8.0.SNAPSHOT jaas-condition-aries_blueprint / 4.1.5 jaas / 4.1.5 aries-blueprint / 4.1.5 odl-infrautils-metrics / 1.4.0.SNAPSHOT wrap / 0.0.0 odl-infrautils-utils / 1.4.0.SNAPSHOT bundle-condition-management / 4.1.5 management / 4.1.5 bundle / 4.1.5 odl-sfc-model / 0.8.0.SNAPSHOT odl-mdsal-models / 0.13.0.SNAPSHOT odl-ovsdb-southbound-api / 1.7.0.SNAPSHOT odl-ovsdb-utils / 1.7.0.SNAPSHOT odl-mdsal-broker / 1.8.0.SNAPSHOT odl-ovsdb-southbound-api / 1.7.0.SNAPSHOT management-condition-aries_blueprint / 4.1.5 management / 4.1.5 aries-blueprint / 4.1.5 odl-yangtools-parser-api / 2.0.5 wrap / 0.0.0 odl-yangtools-common / 2.0.5 odl-yangtools-data / 2.0.5 wrap / 0.0.0 odl-yangtools-data-api / 2.0.5 odl-infrautils-utils / 1.4.0.SNAPSHOT wrap / 0.0.0 odl-guava-23 / 3.1.1 config-condition-shell / 4.1.5 config / 4.1.5 shell / 4.1.5 odl-aaa-password-service / 0.8.0.SNAPSHOT odl-mdsal-binding-base / 2.5.0.SNAPSHOT wrap / 0.0.0 jdbc / 4.1.5 pax-jdbc / 1.1.0 pax-jdbc-config / 1.1.0 kar-condition-shell / 4.1.5 kar / 4.1.5 shell / 4.1.5 odl-mdsal-binding-base / 2.5.0.SNAPSHOT wrap / 0.0.0 odl-yangtools-common / 2.0.5 war / 4.1.5 http / 4.1.5 pax-war / 6.0.9 odl-openflowplugin-app-config-pusher / 0.7.0.SNAPSHOT wrap / 0.0.0 odl-openflowplugin-southbound / 0.7.0.SNAPSHOT scr-condition-management / 4.1.5 scr / 4.1.5 management / 4.1.5 odl-mdsal-model-inventory / 1.8.0.SNAPSHOT odl-mdsal-models / 0.13.0.SNAPSHOT odl-mdsal-binding-runtime / 2.5.0.SNAPSHOT odl-yangtools-util / 2.0.5 wrap / 0.0.0 odl-triemap / 1.0.5 odl-aaa-encryption-service / 0.8.0.SNAPSHOT wrap / 0.0.0 odl-mdsal-broker / 1.8.0.SNAPSHOT ssh / 4.1.5 jaas / 4.1.5 shell / 4.1.5 odl-mdsal-broker / 1.8.0.SNAPSHOT odl-mdsal-remoterpc-connector / 1.8.0.SNAPSHOT odl-mdsal-distributed-datastore / 1.8.0.SNAPSHOT odl-mdsal-dom-api / 2.5.0.SNAPSHOT odl-yangtools-data-api / 2.0.5 odl-mdsal-common / 2.5.0.SNAPSHOT odl-ovsdb-southbound-api / 1.7.0.SNAPSHOT odl-mdsal-models / 0.13.0.SNAPSHOT wrap / 0.0.0 odl-openflowplugin-app-topology / 0.7.0.SNAPSHOT wrap / 0.0.0 odl-openflowplugin-southbound / 0.7.0.SNAPSHOT pax-jetty-condition-pax_http_whiteboard / 9.3.21.v20170918 pax-http-whiteboard / 6.0.9 pax-jetty / 9.3.21.v20170918 scr-condition-shell / 4.1.5 scr / 4.1.5 shell / 4.1.5 system-condition-shell / 4.1.5 system / 4.1.5 shell / 4.1.5 bundle-condition-shell / 4.1.5 bundle / 4.1.5 shell / 4.1.5 odl-sfc-provider-rest / 0.8.0.SNAPSHOT odl-restconf / 1.8.0.SNAPSHOT odl-sfc-provider / 0.8.0.SNAPSHOT pax-jetty-condition-management / 9.3.21.v20170918 pax-jetty / 9.3.21.v20170918 management / 4.1.5 odl-restconf-base / 1.8.0.SNAPSHOT wrap / 0.0.0 odl-restconf-common / 1.8.0.SNAPSHOT shell-compat / 4.1.5 aries-blueprint / 4.1.5 shell / 4.1.5 aries-blueprint-condition-bundle / 4.1.5 aries-blueprint / 4.1.5 bundle / 4.1.5 odl-infrautils-ready / 1.4.0.SNAPSHOT wrap / 0.0.0 odl-guava-23 / 3.1.1 odl-infrautils-utils / 1.4.0.SNAPSHOT odl-mdsal-distributed-datastore / 1.8.0.SNAPSHOT wrap / 0.0.0 odl-mdsal-broker-local / 1.8.0.SNAPSHOT odl-mdsal-clustering-commons / 1.8.0.SNAPSHOT odl-controller-mdsal-common / 1.8.0.SNAPSHOT odl-mdsal-binding-runtime / 2.5.0.SNAPSHOT odl-config-api / 0.9.0.SNAPSHOT odl-mdsal-common / 2.5.0.SNAPSHOT odl-mdsal-singleton-dom / 2.5.0.SNAPSHOT odl-mdsal-singleton-common / 2.5.0.SNAPSHOT odl-mdsal-eos-dom / 2.5.0.SNAPSHOT odl-mdsal-common / 2.5.0.SNAPSHOT odl-infrautils-caches / 1.4.0.SNAPSHOT wrap / 0.0.0 odl-guava-23 / 3.1.1 odl-infrautils-utils / 1.4.0.SNAPSHOT odl-mdsal-remoterpc-connector / 1.8.0.SNAPSHOT odl-mdsal-broker-local / 1.8.0.SNAPSHOT odl-mdsal-clustering-commons / 1.8.0.SNAPSHOT odl-akka-leveldb-0.7 / 3.1.0 odl-mdsal-eos-binding / 2.5.0.SNAPSHOT wrap / 0.0.0 odl-mdsal-binding-dom-adapter / 2.5.0.SNAPSHOT odl-mdsal-eos-dom / 2.5.0.SNAPSHOT 51b3d2b9-436d-4263-80a8-dbb1cfb39b85 / 0.0.0 ssh / 4.1.5 framework / 4.1.5 standard / 4.1.5 odl-genius-tools / 0.5.0.SNAPSHOT wrap / 0.0.0 odl-mdsal-broker / 1.8.0.SNAPSHOT odl-infrautils-metrics / 1.4.0.SNAPSHOT odl-config-netty / 0.9.0.SNAPSHOT wrap / 0.0.0 odl-config-startup / 0.9.0.SNAPSHOT odl-config-netty-config-api / 0.9.0.SNAPSHOT odl-aaa-web / 0.8.0.SNAPSHOT odl-guava-23 / 3.1.1 odl-karaf-feat-jetty / 3.1.0 odl-yangtools-util / 2.0.5 odl-aaa-cert / 0.8.0.SNAPSHOT wrap / 0.0.0 odl-mdsal-broker / 1.8.0.SNAPSHOT odl-aaa-encryption-service / 0.8.0.SNAPSHOT odl-config-core / 0.9.0.SNAPSHOT odl-config-util / 0.9.0.SNAPSHOT wrap / 0.0.0 odl-mdsal-binding-runtime / 2.5.0.SNAPSHOT odl-controller-model-topology / 1.8.0.SNAPSHOT odl-mdsal-models / 0.13.0.SNAPSHOT odl-mdsal-binding-runtime / 2.5.0.SNAPSHOT odl-mdsal-model-inventory / 1.8.0.SNAPSHOT odl-sfc-ovs / 0.8.0.SNAPSHOT wrap / 0.0.0 odl-ovsdb-southbound-impl / 1.7.0.SNAPSHOT odl-sfc-provider-rest / 0.8.0.SNAPSHOT odl-config-util / 0.9.0.SNAPSHOT wrap / 0.0.0 odl-yangtools-data-api / 2.0.5 odl-config-api / 0.9.0.SNAPSHOT odl-infrautils-counters / 1.4.0.SNAPSHOT wrap / 0.0.0 odl-guava-23 / 3.1.1 odl-infrautils-utils / 1.4.0.SNAPSHOT odl-config-netty-config-api / 0.9.0.SNAPSHOT wrap / 0.0.0 odl-netty-4 / 3.1.0 odl-config-api / 0.9.0.SNAPSHOT odl-infrautils-jobcoordinator / 1.4.0.SNAPSHOT wrap / 0.0.0 odl-guava-23 / 3.1.1 odl-infrautils-utils / 1.4.0.SNAPSHOT odl-infrautils-metrics / 1.4.0.SNAPSHOT odl-openflowplugin-flow-services / 0.7.0.SNAPSHOT odl-openflowplugin-app-reconciliation-framework / 0.7.0.SNAPSHOT odl-openflowplugin-app-topology / 0.7.0.SNAPSHOT odl-openflowplugin-app-forwardingrules-manager / 0.7.0.SNAPSHOT odl-openflowplugin-app-config-pusher / 0.7.0.SNAPSHOT odl-ovsdb-southbound-impl / 1.7.0.SNAPSHOT wrap / 0.0.0 odl-mdsal-broker / 1.8.0.SNAPSHOT odl-ovsdb-library / 1.7.0.SNAPSHOT odl-ovsdb-southbound-api / 1.7.0.SNAPSHOT odl-mdsal-binding-api / 2.5.0.SNAPSHOT odl-mdsal-binding-base / 2.5.0.SNAPSHOT wrap / 0.0.0 odl-yangtools-data-api / 2.0.5 odl-mdsal-common / 2.5.0.SNAPSHOT odl-mdsal-clustering-commons / 1.8.0.SNAPSHOT odl-mdsal-broker-local / 1.8.0.SNAPSHOT odl-akka-persistence-2.5 / 3.1.0 odl-akka-system-2.5 / 3.1.0 odl-akka-clustering-2.5 / 3.1.0 odl-mdsal-binding-dom-adapter / 2.5.0.SNAPSHOT wrap / 0.0.0 odl-mdsal-binding-runtime / 2.5.0.SNAPSHOT odl-mdsal-dom-api / 2.5.0.SNAPSHOT odl-mdsal-binding-api / 2.5.0.SNAPSHOT odl-openflowplugin-nsf-model / 0.7.0.SNAPSHOT odl-mdsal-models / 0.13.0.SNAPSHOT odl-mdsal-model-inventory / 1.8.0.SNAPSHOT odl-openflowjava-protocol / 0.7.0.SNAPSHOT odl-controller-model-topology / 1.8.0.SNAPSHOT odl-openflowjava-protocol / 0.7.0.SNAPSHOT odl-mdsal-models / 0.13.0.SNAPSHOT odl-mdsal-binding-base / 2.5.0.SNAPSHOT wrap / 0.0.0 odl-netty-4 / 3.1.0 odl-sfc-provider / 0.8.0.SNAPSHOT wrap / 0.0.0 odl-mdsal-broker / 1.8.0.SNAPSHOT odl-genius-api / 0.5.0.SNAPSHOT odl-sfc-model / 0.8.0.SNAPSHOT odl-sfc-genius / 0.8.0.SNAPSHOT wrap / 0.0.0 odl-mdsal-broker / 1.8.0.SNAPSHOT odl-sfc-provider / 0.8.0.SNAPSHOT odl-genius-api / 0.5.0.SNAPSHOT odl-ovsdb-library / 1.7.0.SNAPSHOT odl-mdsal-models / 0.13.0.SNAPSHOT wrap / 0.0.0 odl-mdsal-broker / 1.8.0.SNAPSHOT odl-aaa-cert / 0.8.0.SNAPSHOT odl-sfc-openflow-renderer / 0.8.0.SNAPSHOT wrap / 0.0.0 odl-sfc-genius / 0.8.0.SNAPSHOT odl-openflowplugin-flow-services / 0.7.0.SNAPSHOT odl-openflowplugin-nxm-extensions / 0.7.0.SNAPSHOT odl-sfc-ovs / 0.8.0.SNAPSHOT odl-infrautils-diagstatus / 1.4.0.SNAPSHOT odl-infrautils-ready / 1.4.0.SNAPSHOT wrap / 0.0.0 odl-guava-23 / 3.1.1 odl-karaf-feat-jetty / 3.1.0 odl-infrautils-utils / 1.4.0.SNAPSHOT odl-mdsal-binding-runtime / 2.5.0.SNAPSHOT odl-mdsal-binding-base / 2.5.0.SNAPSHOT wrap / 0.0.0 odl-yangtools-parser / 2.0.5 odl-yangtools-data / 2.0.5 odl-javassist-3 / 3.1.0 odl-mdsal-dom-broker / 2.5.0.SNAPSHOT wrap / 0.0.0 odl-lmax-3 / 3.1.0 odl-yangtools-parser / 2.0.5 odl-mdsal-dom-api / 2.5.0.SNAPSHOT odl-yangtools-data / 2.0.5 odl-openflowplugin-southbound / 0.7.0.SNAPSHOT wrap / 0.0.0 odl-mdsal-broker / 1.8.0.SNAPSHOT odl-openflowplugin-nsf-model / 0.7.0.SNAPSHOT odl-openflowplugin-libraries / 0.7.0.SNAPSHOT odl-infrautils-diagstatus / 1.4.0.SNAPSHOT odl-sfc-scf-openflow / 0.8.0.SNAPSHOT wrap / 0.0.0 odl-sfc-genius / 0.8.0.SNAPSHOT odl-openflowplugin-flow-services / 0.7.0.SNAPSHOT odl-openflowplugin-nxm-extensions / 0.7.0.SNAPSHOT odl-sfc-openflow-renderer / 0.8.0.SNAPSHOT odl-sfc-ovs / 0.8.0.SNAPSHOT odl-config-persister / 0.9.0.SNAPSHOT wrap / 0.0.0 odl-mdsal-binding-runtime / 2.5.0.SNAPSHOT odl-config-manager / 0.9.0.SNAPSHOT odl-config-manager-facade-xml / 0.9.0.SNAPSHOT odl-config-api / 0.9.0.SNAPSHOT odl-config-core / 0.9.0.SNAPSHOT odl-restconf-common / 1.8.0.SNAPSHOT odl-mdsal-models / 0.13.0.SNAPSHOT odl-jackson-2.8 / 3.1.0 wrap / 0.0.0 odl-yangtools-export / 2.0.5 odl-mdsal-broker / 1.8.0.SNAPSHOT odl-karaf-feat-jetty / 3.1.0 odl-aaa-shiro / 0.8.0.SNAPSHOT odl-aaa-shiro / 0.8.0.SNAPSHOT wrap / 0.0.0 odl-aaa-web / 0.8.0.SNAPSHOT odl-aaa-password-service / 0.8.0.SNAPSHOT odl-karaf-feat-jetty / 3.1.0 odl-karaf-feat-jdbc / 3.1.0 odl-aaa-api / 0.8.0.SNAPSHOT odl-aaa-cert / 0.8.0.SNAPSHOT odl-jolokia / 1.11.0.SNAPSHOT odl-aaa-encryption-service / 0.8.0.SNAPSHOT odl-mdsal-broker-local / 1.8.0.SNAPSHOT odl-mdsal-models / 0.13.0.SNAPSHOT wrap / 0.0.0 odl-lmax-3 / 3.1.0 odl-config-startup / 0.9.0.SNAPSHOT odl-yangtools-codec / 2.0.5 odl-mdsal-binding-dom-adapter / 2.5.0.SNAPSHOT odl-config-netty / 0.9.0.SNAPSHOT odl-controller-mdsal-common / 1.8.0.SNAPSHOT odl-mdsal-eos-binding / 2.5.0.SNAPSHOT odl-mdsal-dom / 2.5.0.SNAPSHOT odl-mdsal-singleton-dom / 2.5.0.SNAPSHOT odl-mdsal-eos-dom / 2.5.0.SNAPSHOT odl-genius-api / 0.5.0.SNAPSHOT odl-mdsal-broker / 1.8.0.SNAPSHOT odl-genius-tools / 0.5.0.SNAPSHOT odl-infrautils-inject / 1.4.0.SNAPSHOT odl-infrautils-jobcoordinator / 1.4.0.SNAPSHOT odl-ovsdb-southbound-api / 1.7.0.SNAPSHOT odl-openflowplugin-libraries / 0.7.0.SNAPSHOT odl-ovsdb-hwvtepsouthbound-api / 1.7.0.SNAPSHOT odl-openflowplugin-nxm-extensions / 0.7.0.SNAPSHOT odl-mdsal-models / 0.13.0.SNAPSHOT odl-infrautils-caches / 1.4.0.SNAPSHOT odl-openflowplugin-nsf-model / 0.7.0.SNAPSHOT odl-infrautils-counters / 1.4.0.SNAPSHOT odl-ovsdb-utils / 1.7.0.SNAPSHOT standard / 4.1.5 ssh / 4.1.5 service / 4.1.5 log / 4.1.5 jaas / 4.1.5 management / 4.1.5 aries-blueprint / 4.1.5 feature / 4.1.5 kar / 4.1.5 diagnostic / 4.1.5 package / 4.1.5 wrap / 0.0.0 deployer / 4.1.5 instance / 4.1.5 config / 4.1.5 system / 4.1.5 shell-compat / 4.1.5 bundle / 4.1.5 shell / 4.1.5
|
| Comment by Faseela K [ 12/Jun/18 ] |
|
seen in odl-netvirt-openstack as well : Console Logs: https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/autorelease-release-oxygen/319 Jenkins Build: https://jenkins.opendaylight.org/releng/job/autorelease-release-oxygen/319/ |
| Comment by Michael Vorburger [ 14/Jun/18 ] |
|
odlparent 3.1.2 will include c/72400; I'm sure it will get released soon-ish, then projects can bump to it, and this shouldn't be seen anymore. |
| Comment by Michael Vorburger [ 25/Jul/18 ] |
|
We have been seen again internally (not in an SFT but at runtime), but this time the attached karaf.log shows this: 2018-07-24T22:34:58,531 | WARN | pool-18-thread-2 | core | 85 - org.apache.aries.jmx.core - 1.1.7 | Task rejected for JMX Notification dispatch of event [org.osgi.framework.ServiceEvent[source={javax.management.MBeanServer}={service.id=132, service.bundleid=126, service.scope=singleton}]] - Dispatcher may have been shutdown
2018-07-24T22:34:58,559 | WARN | pool-18-thread-2 | Activator | 126 - org.apache.karaf.management.server - 4.1.5 | Error starting activator
java.rmi.server.ExportException: Port already in use: 1099; nested exception is:
java.net.BindException: Address already in use (Bind failed)
at sun.rmi.transport.tcp.TCPTransport.listen(TCPTransport.java:346) [?:?]
at sun.rmi.transport.tcp.TCPTransport.exportObject(TCPTransport.java:254) [?:?]
at sun.rmi.transport.tcp.TCPEndpoint.exportObject(TCPEndpoint.java:411) [?:?]
at sun.rmi.transport.LiveRef.exportObject(LiveRef.java:147) [?:?]
at sun.rmi.server.UnicastServerRef.exportObject(UnicastServerRef.java:236) [?:?]
at sun.rmi.registry.RegistryImpl.setup(RegistryImpl.java:213) [?:?]
at sun.rmi.registry.RegistryImpl.<init>(RegistryImpl.java:173) [?:?]
at sun.rmi.registry.RegistryImpl.<init>(RegistryImpl.java:144) [?:?]
at java.rmi.registry.LocateRegistry.createRegistry(LocateRegistry.java:239) [?:?]
at org.apache.karaf.management.RmiRegistryFactory.init(RmiRegistryFactory.java:126) [126:org.apache.karaf.management.server:4.1.5]
at org.apache.karaf.management.internal.Activator.doStart(Activator.java:104) [126:org.apache.karaf.management.server:4.1.5]
at org.apache.karaf.util.tracker.BaseActivator.run(BaseActivator.java:242) [126:org.apache.karaf.management.server:4.1.5]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:?]
at java.lang.Thread.run(Thread.java:748) [?:?]
Caused by: java.net.BindException: Address already in use (Bind failed)
at java.net.PlainSocketImpl.socketBind(Native Method) ~[?:?]
at java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:387) ~[?:?]
at java.net.ServerSocket.bind(ServerSocket.java:375) ~[?:?]
at java.net.ServerSocket.<init>(ServerSocket.java:237) ~[?:?]
at org.apache.karaf.management.RmiRegistryFactory$KarafServerSocketFactory.createServerSocket(RmiRegistryFactory.java:181) ~[?:?]
at sun.rmi.transport.tcp.TCPEndpoint.newServerSocket(TCPEndpoint.java:666) ~[?:?]
at sun.rmi.transport.tcp.TCPTransport.listen(TCPTransport.java:335) ~[?:?]
... 16 more
|
