https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/RELENG-114 releng <p>red dots are what we see on infra aborts. However, there are also other instances<br/> when we will see red dots that are not red dots. For example, if some bug creeps<br/> in that bloats our ODL logs to such a large size that we cannot archive them  and<br/> and jenkins gives up (speaking from experience).</p> <p>this task is for someone to take a reasonable sample of the recent red dots in<br/> CSIT jobs (preferabbly netvirt, openflowplugin and controller) and report back<br/> the percentage of infra aborts vs not.</p> <p>Based on this data, we can decide how smart we want to get with our bat signal<br/> deciding when to sound off. We don't want to create so much noise that it's annoying<br/> meaningless to the admins that will be expected to watch this.</p> RELENG-114

Analyze CSIT Red Dots

Sub-task RELENG-113 Medium Closed Done Jamo Luhrsen Jamo Luhrsen Fri, 4 May 2018 00:53:54 +0000 Fri, 1 Jun 2018 19:07:34 +0000 Fri, 1 Jun 2018 19:07:34 +0000 0 1 <p>some analysis:</p> <p> <br/> <a href="https://jenkins.opendaylight.org/releng/job/netvirt-csit-1node-openstack-pike-upstream-stateful-fluorine" class="external-link" target="_blank" rel="nofollow noopener">this job</a> had 4 red dots in 30 runs over the course of a month. Only one <a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-openstack-pike-upstream-stateful-fluorine/87/console-timestamp.log.gz" class="external-link" target="_blank" rel="nofollow noopener">1</a> red dot was an infra abort.</p> <p><a href="https://jenkins.opendaylight.org/releng/job/netvirt-csit-1node-openstack-pike-upstream-stateful-oxygen/" class="external-link" target="_blank" rel="nofollow noopener">this job</a> had 13 runs over a month, 2 of which were red dots. One <a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-openstack-pike-upstream-stateful-oxygen/271/console-timestamp.log.gz" class="external-link" target="_blank" rel="nofollow noopener">1</a> of those red dots was an infra issue.</p> <p><a href="https://jenkins.opendaylight.org/releng/user/jluhrsen/my-views/view/netvirt%20csit/job/netvirt-csit-1node-openstack-queens-upstream-stateful-fluorine/" class="external-link" target="_blank" rel="nofollow noopener">this job</a> had 30 runs over a four day period with two red dots. Neither red dot was a problem with infra.</p> <p><a href="https://jenkins.opendaylight.org/releng/user/jluhrsen/my-views/view/netvirt%20csit/job/netvirt-csit-1node-openstack-queens-upstream-stateful-oxygen/" class="external-link" target="_blank" rel="nofollow noopener">this job</a> also with 30 runs over a four day period had 3 red dots, none of which were infra related.</p> <p>To summarize, that's aprox 120 jobs with 11 red dots, but only 2 were infra related. So, essentially we have a 1.6% rate of infra<br/> aborts. Also, the 2 infra aborts we saw were not some indication of infra being broken and affecting jobs running afterward.</p> <p>I'm not sure it's justifiable to spend time on automation to warn the right people if we get an abort, unless someone has<br/> the extra cycles or there is some other problem I'm missing. This is not to say we didn't have major issues in the past when<br/> something like this would have been very useful.</p> Development Rank 0|i03e8v: