[CONTROLLER-1747] Leader movement longevity job is unstable

Tue, 25 Jul 2023 08:24:45 +0000

This weekend, job controller-csit-3node-ddb-expl-lead-movement-longevity-only-carbon failed to finish correctly on RelEng. Looking at console output [0], the robot execution passed, but during post-processing the Robot data, connection to the Robot VM was lost.

An attempt at reproducing this behavior on Sandbox has failed, in a sense that ReadTimeout happened after 11 hours and examining console output [1] it is clear that connections to ODL VMs have been lost.

The job has never shown this types of behavior before, so this is a regression. Both types can be explained if there is an exception generating long restconf outputs and karaf log. ODL VM failures happens when disk is full, Robot VM when output.xml is too large.

Unfortunately, attempts to reproduce with shorter runtimes on Sandbox are failing so far (meaning the test passes without issues), so the real cause might be something different, including just bad luck with infra in two occasions.

[0] https://jenkins.opendaylight.org/releng/view/controller/job/controller-csit-3node-ddb-expl-lead-movement-longevity-only-carbon/18/console
[1] https://jenkins.opendaylight.org/sandbox/job/controller-csit-3node-ddb-expl-lead-movement-longevity-only-carbon/1/console

OpenDaylight JIRA

[CONTROLLER-1747] Leader movement longevity job is unstable