OpenDaylight JIRA

OpenDaylight JIRA https://jira.opendaylight.org This file is an XML representation of an issue en-us

8.20.10

820010

22-06-2022

[INTTEST-6] Leader election takes too long after shutdown instance starts https://jira.opendaylight.org/browse/INTTEST-6 integration-test <p>This is occurring sporadically in the cluster system test:</p> <p><a href="https://jenkins.opendaylight.org/releng/view/CSIT-3node/job/controller-csit-3node-clustering-only-beryllium/565/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/CSIT-3node/job/controller-csit-3node-clustering-only-beryllium/565/</a></p> <p>See karaf attached log excerpt for the instance start. After 5 mins the leader is not elected in this partcular instance.</p> <p>Operating System: All<br/> Platform: All</p> INTTEST-6

Leader election takes too long after shutdown instance starts

Bug Resolved Done Luis Gomez Luis Gomez Tue, 9 Feb 2016 18:13:16 +0000 Thu, 19 Oct 2017 22:33:49 +0000 Thu, 25 Feb 2016 10:46:32 +0000 0 3 <p>Attachment node_start.txt has been added with description: karaf log node start</p> <p>Full karaf log for the failing instance is in odl1_karaf.log.xz file of the mentioned job run:</p> <p><a href="https://jenkins.opendaylight.org/releng/view/CSIT-3node/job/controller-csit-3node-clustering-only-beryllium/565/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/CSIT-3node/job/controller-csit-3node-clustering-only-beryllium/565/</a></p> <p>At the start of "030 Car Failover Crud On New Leader", odl1 (member-1 @ 10.30.11.161) was the leader. The test stops the leader and leadership transitioned to odl3 (member-3 @ 10.30.11.142). The test then does some CRUD with the remaining 2 nodes and then restarts the previous leader odl1. </p> <p>odl1 started up at:</p> <p>2016-02-08 13:28:10,994 | INFO | ult-dispatcher-2 | kka://opendaylight-cluster-data) | 129 - com.typesafe.akka.slf4j - 2.3.14 | Cluster Node <span class="error">[akka.tcp://opendaylight-cluster-data@10.30.11.161:2550]</span> - Starting up...</p> <p>and then joined itself and declared itself as cluster leader at:</p> <p>2016-02-08 13:28:16,074 | INFO | lt-dispatcher-23 | kka://opendaylight-cluster-data) | 129 - com.typesafe.akka.slf4j - 2.3.14 | Cluster Node <span class="error">[akka.tcp://opendaylight-cluster-data@10.30.11.161:2550]</span> - Node <span class="error">[akka.tcp://opendaylight-cluster-data@10.30.11.161:2550]</span> is JOINING, roles <span class="error">[member-1]</span><br/> 2016-02-08 13:28:17,062 | INFO | lt-dispatcher-23 | kka://opendaylight-cluster-data) | 129 - com.typesafe.akka.slf4j - 2.3.14 | Cluster Node <span class="error">[akka.tcp://opendaylight-cluster-data@10.30.11.161:2550]</span> - Leader is moving node <span class="error">[akka.tcp://opendaylight-cluster-data@10.30.11.161:2550]</span> to <span class="error">[Up]</span></p> <p>So odl1 didn't connect to odl3 and formed an island with itself as leader. This is b/c odl1 is the first seed node and akka has special behavior wrt first seed node. It will first try to connect to another seed node and, failing that, it declares itself leader. This is governed by the seed-node-timeout setting which is 5s by default. We had seen this issue before and had changed the seed-node-timeout to 12s in the akka.conf we ship. It looks like the tests are still using the default 5s judging from the timestamps above. I believe the deployment script copies over its own akka.conf so akka.conf template needs to be updated to the latest. It would be nice if the tests could get the latest akka.conf from the build and substitute the appropriate settings on the fly so we don't keep running into issues with a stale akka.conf.</p> <p>Wrt stale akka.conf, this will happen in real environments on upgrade. What we need is a factory akka.conf that we overwrite on feature install and a custom akka.conf that users modify and we preserve. The custom akka.conf would be overlaid/merged with the factory akka.conf. We can do this for Bo and maybe backport to Be SR1.</p> <p>Good comments Tom, I think we will have to change the way the cluster files are fed into the deploy job, moving this to integration bugzilla.</p> <p>For now you can just update the akka.conf template to match what we ship in the controller.</p> <p>Going forward we can implement the factory/custom approach I mentioned so the tests would pickup factory changes automatically. So you don't need to invest any time to do something like that in the test framework. The akka.conf doesn't change that often anyway.</p> <p>Right, short term (today) we will update the templates in integration. Medium term (after Be release) we will update deploy scripts to use distribution scripts.</p> <p>So after looking at the cluster deploy scripts, they use (now I rememeber shaleen changed this) the right templates from the distribution itself:</p> <p>AKKACONF=/tmp/${BUNDLEFOLDER}/configuration/initial/akka.conf<br/> MODULESCONF=/tmp/${BUNDLEFOLDER}/configuration/initial/modules.conf<br/> MODULESHARDSCONF=/tmp/${BUNDLEFOLDER}/configuration/initial/module-shards.conf</p> <p>This means the issues is still there with the right cluster templates, so returning this bug to controller.</p> <p>I don't think it is. The seed-node-timeout is set to 12s in the akk.conf we ship. The timestamps from the log indicate akka only waited 5s (13:28:10,994 -> 13:28:16,074 ~ 5 sec). I would suggest checking the seed-node-timeout setting in the akka.conf on each node after deploy.</p> <p>You are right, after deeper look we are still using integration templates for this. So changing the bug to me. Sorry for the ping-pong.</p> <p>BR/Luis</p> <p>NP - as long as we get it right in the end <img class="emoticon" src="https://jira.opendaylight.org/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>(In reply to Luis Gomez from comment #9)<br/> > You are right, after deeper look we are still using integration templates<br/> > for this. So changing the bug to me. Sorry for the ping-pong.<br/> > <br/> > BR/Luis</p> <p>Tom, can you please review this:</p> <p><a href="https://git.opendaylight.org/gerrit/#/c/34432" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/34432</a></p> <p>> <a href="https://git.opendaylight.org/gerrit/#/c/34432" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/34432</a></p> <p>Merged, so setting as FIXED.</p> Development Rank 0|i01lmf: