[ODLPARENT-144] SingleFeatureTest failures due to ScrServiceMBean

Relates — Fri, 15 Mar 2019 17:42:54 +0000

We're suddenly regularly seeing this SFT failure:

Running org.opendaylight.odlparent.featuretest.SingleFeatureTest
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 332.267 sec <<< FAILURE! - in org.opendaylight.odlparent.featuretest.SingleFeatureTest
installFeatureCatchAndLog(org.opendaylight.odlparent.featuretest.SingleFeatureTest)[repoUrl: file:/w/workspace/neutron-maven-verify-oxygen-mvn33-openjdk8/features/production/odl-neutron-transcriber/target/feature/feature.xml, Feature: odl-neutron-transcriber 0.10.0.SNAPSHOT]  Time elapsed: 331.59 sec  <<< ERROR!
org.awaitility.core.ConditionTimeoutException: Condition with alias 'checkBundleDiagInfos' didn't complete within 300 seconds because lambda expression in org.opendaylight.odlparent.bundlestest.lib.TestBundleDiag: expected system either ready with all bundles Active, or Stopping or Failure (but not still booting in GracePeriod, Waiting, Starting, Unknown;but just Resolved and some exceptional Installed OK) but was <diag: Booting {Installed=0, Resolved=6, Unknown=0, GracePeriod=0, Waiting=1, Starting=0, Active=351, Stopping=0, Failure=0}
1. NOK org.apache.karaf.scr.management:4.1.3: OSGi state = Active, Karaf bundleState = Waiting, due to: Declarative Services
ScrServiceMBean (2)
  missing references: mBeanServer
>.
	at org.awaitility.core.ConditionAwaiter.await(ConditionAwaiter.java:136)
	at org.awaitility.core.AbstractHamcrestCondition.await(AbstractHamcrestCondition.java:89)
	at org.awaitility.core.ConditionFactory.until(ConditionFactory.java:840)
	at org.awaitility.core.ConditionFactory.until(ConditionFactory.java:619)
	at org.opendaylight.odlparent.bundlestest.lib.TestBundleDiag.checkBundleDiagInfos(TestBundleDiag.java:71)
	at org.opendaylight.odlparent.bundlestest.lib.TestBundleDiag.checkBundleDiagInfos(TestBundleDiag.java:54)
	at org.opendaylight.odlparent.featuretest.SingleFeatureTest.installFeature(SingleFeatureTest.java:404)
	at org.opendaylight.odlparent.featuretest.SingleFeatureTest.installFeatureCatchAndLog(SingleFeatureTest.java:355)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.lang.Thread.run(Thread.java:748)


Results :

Tests in error: 
  Condition with alias 'checkBundleDiagInfos' didn't complete within 300 seconds because lambda expression in org.opendaylight.odlparent.bundlestest.lib.TestBundleDiag: expected system either ready with all bundles Active, or Stopping or Failure (but not still booting in GracePeriod, Waiting, Starting, Unknown;but just Resolved and some exceptional Installed OK) but was <diag: Booting {Installed=0, Resolved=6, Unknown=0, GracePeriod=0, Waiting=1, Starting=0, Active=351, Stopping=0, Failure=0}
1. NOK org.apache.karaf.scr.management:4.1.3: OSGi state = Active, Karaf bundleState = Waiting, due to: Declarative Services
ScrServiceMBean (2)
  missing references: mBeanServer

e.g. here https://jenkins.opendaylight.org/releng/job/neutron-maven-verify-oxygen-mvn33-openjdk8/165/console (for https://git.opendaylight.org/gerrit/#/c/69397/, but totally unrelated) and here https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/autorelease-release-oxygen/215/console-timestamp.log.gz

We got two in the past week, and at least I don't recall have seen this one before - have others?

They are intermittent and disappear after "recheck" - but ideally should never happen.

Should we just increase the SFT time-out? Should we whitelist the "org.apache.karaf.scr.management" bundle in SFT, as in "we REALLY don't care if some Karaf Service Component Runtime (SCR) OSGi Declarative Services (DS) stupid JMX monitoring bean failed to come up for some reason, as long as all of our other ODL features installed correctly" ?

The odlparent project would certainly welcome anyone's contributions for this.

OpenDaylight JIRA

[ODLPARENT-144] SingleFeatureTest failures due to ScrServiceMBean