Operating System: All
Platform: All

sorry, I do not have a full karaf.log. The console output from one of jobs is attached which gives the first and last 500K bytes. Note, there are a ton of AuthenticationExceptions, but those are expected as it's our polling on restconf by our bringup script

Attachment 5346.karaf.log.gz has been added with description: first and last 500K bytes of karaf.log

These failures do not happen with Beryllium snapshot builds, but they do not happen when testing RC3 on Sandbox [0] either.

Common denominator is netconf connector. The jobs either install odl-netconf-connector-all (in job definition), or they install odl-netconf-connector (via odl-usc-channel).

The error message mentions odl-netconf-client feature, while global-worker-group is defined in 00-netty.xml config file of odl-config-netty feature (which is not listed as dependency of odl-netconf-client).

Probably, the 00-netty.xml is pulled in by some other feature being installed, and it depends on system performance whether ODL detects the transient inconsistency or not.

Change 34675 [1] contributed to address the dependency issue, but there may be more hidden issues of this type, see comments of ODLPARENT-12.

[0] https://jenkins.opendaylight.org/sandbox/job/netconf-csit-1node-userfeatures-only-beryllium/1
[1] https://git.opendaylight.org/gerrit/34675

Thanks Vratko!

Is this related?

> 2016-02-13 13:44:49,924 | INFO | Event Dispatcher | YangTextSchemaContextResolver | 69 - org.opendaylight.yangtools.yang-parser-impl - 0.8.0.Beryllium | Provided module name /META-INF/yang/threadpool.yang@0000-00-00.yang does not match actual text threadpool@2013-04-09.yang, corrected

> https://git.opendaylight.org/gerrit/34675

https://jenkins.opendaylight.org/releng/view/netconf/job/netconf-csit-1node-userfeatures-only-beryllium/206
is queued and it shall verify this Change does not break snapshot builds.

But I do not see an easy way to verify its supposed effect on autorelease builds, without actually making new autorelease build with the Change merged.

> Is this related?

No, that is a warning from new Yangtools parser, after the automatic source name conversion everything works.

This means we are re-spinning the release right?

there are other "ready-to-submit" patches out there as well.

This means we are re-spinning the release right?

there are other "ready-to-submit" patches out there as well.

> we are re-spinning the release right?

+1

Severity: Could you elaborate on the severity of this bug? Is this a BLOCKER such that we cannot release Beryllium without it? Is there a workaround such that we can write a release note and fix in Beryllium SR1?
Testing: Could you also elaborate on the testing or this patch? How extensively has this patch been tested? Is it covered by any unit tests or system tests?
Impact: Does this fix impact any dependent projects?

(In reply to A H from comment #10)
> Severity: Could you elaborate on the severity of this bug? Is this a
> BLOCKER such that we cannot release Beryllium without it? Is there a
> workaround such that we can write a release note and fix in Beryllium SR1?

I don't know of any workaround, but I didn't really look. IMO this is a
release blocker as it prevents restconf from coming up when specific
user facing features are installed.

> Testing: Could you also elaborate on the testing or this patch? How
> extensively has this patch been tested? Is it covered by any unit tests or
> system tests?

There are the four system test jobs (initial comment) that will give us
good confidence that the patch gets us past this bug.

> Impact: Does this fix impact any dependent projects?

not sure. Hopefully Vratko can add more info here.

(In reply to Jamo Luhrsen from comment #11)
> (In reply to A H from comment #10)
> > Severity: Could you elaborate on the severity of this bug? Is this a
> > BLOCKER such that we cannot release Beryllium without it? Is there a
> > workaround such that we can write a release note and fix in Beryllium SR1?
>
> I don't know of any workaround, but I didn't really look. IMO this is a
> release blocker as it prevents restconf from coming up when specific
> user facing features are installed.
>
> > Testing: Could you also elaborate on the testing or this patch? How
> > extensively has this patch been tested? Is it covered by any unit tests or
> > system tests?
>
> There are the four system test jobs (initial comment) that will give us
> good confidence that the patch gets us past this bug.

I would also agree that the four CSIT that failed would cover the testing aspects of this patch.

> > Impact: Does this fix impact any dependent projects?
>
> not sure. Hopefully Vratko can add more info here.

This fix would impact projects whose CSIT jobs failed originally. We will know more once we rerun these jobs after merging the fix.

> Could you elaborate on the severity of this bug?

When user installs a feature related to netconf-connector, ODL may break so that even basic Restconf does not work. This is a blocker unless a workaround is applied.

> How extensively has this patch been tested?

This Bug is not reproducible on Sandbox or on Snapshot builds, probably because a timing issue. The Bug was reproduced on Releng with original autorelease build.
The fix was only proven to not make things worse. Even when it improves the code, next respin may fail on similar issue in other parts of code.

> Is there a workaround such that we can write a release note and fix in Beryllium SR1?

If user installs odl-config-netty feature prior anything related to netconf-connector, ODL should work.
Due to difficulties with reproducing this Bug, it is not clear whether this workaround indeed works.

> Is it covered by any unit tests or system tests?

No specific system tests, apart the 4 csit jobs that were reported.
No unit tests for this Bug exist right now. Comments in ODLPARENT-12 may enable creation of such unit tests, but probably not before Beryllium Release.

> Does this fix impact any dependent projects?

No, projects were already expecting netconf-connector to install all its dependencies.

The bgpcep job triggered with autorelease URL on RelEng Jenkins works again [0].
Setting to FIXED.

[0] https://jenkins.opendaylight.org/releng/job/bgpcep-csit-1node-userfeatures-only-beryllium/1094/console

Have we been able to verify the fix for this bug?

(In reply to A H from comment #15)
> Have we been able to verify the fix for this bug?

using the CSIT jobs with RC3 we can mark this verified.