https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/CONTROLLER-1768 controller <p>This was seen in a netvirt 3node job:</p> <p><a href="https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-3node-openstack-ocata-upstream-stateful-carbon/118/log.html.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-3node-openstack-ocata-upstream-stateful-carbon/118/log.html.gz</a></p> <p>The job has the karaf logs, and in the log for ODL2 I saw this:</p> <p>2017-09-08 12:43:45,101 | ERROR | Event Dispatcher | AbstractDataStore | 203 - org.opendaylight.controller.sal-distributed-datastore - 1.5.2.SNAPSHOT | Shard leaders failed to settle in 90 seconds, giving up</p> <p>The system tests were pasing ok until this point, after which a lot<br/> of failures were seen.</p> <p>Operating System: All<br/> Platform: All</p> CONTROLLER-1768

SyncStatus stays false for more than 5minutes after bringing 2 of 3 nodes down and back up.

Bug High Verified Cannot Reproduce Unassigned Jamo Luhrsen csit:3node Fri, 8 Sep 2017 23:10:18 +0000 Wed, 18 Jul 2018 21:35:16 +0000 Wed, 18 Jul 2018 21:35:09 +0000 Carbon Oxygen Oxygen SR3 Fluorine Oxygen SR3 clustering 0 7 <p>It seems that shard leaders weren't resolved after 90 sec but eventually did around 2017-09-08 12:49, eg</p> <p> INFO | rd-dispatcher-30 | ShardManager | 203 - org.opendaylight.controller.sal-distributed-datastore - 1.5.2.SNAPSHOT | shard-manager-config: Received role changed for member-2-shard-inventory-config from Candidate to Leader</p> <p>I'm not sure what the expectations of the test are but I suspect this is similar to <a href="https://jira.opendaylight.org/browse/CONTROLLER-1751" title="Sporadic cluster failure when member is restarted in OF cluster test" class="issue-link" data-issue-key="CONTROLLER-1751"><del>CONTROLLER-1751</del></a> where node connectivity occasionally gets delayed in the test environment for some reason (hypothesis is overload in the environment).</p> <p>It seems that shard leaders weren't resolved after 90 sec but eventually did around 2017-09-08 12:49, eg</p> <p> INFO | rd-dispatcher-30 | ShardManager | 203 - org.opendaylight.controller.sal-distributed-datastore - 1.5.2.SNAPSHOT | shard-manager-config: Received role changed for member-2-shard-inventory-config from Candidate to Leader</p> <p>I'm not sure what the expectations of the test are but I suspect this is similar to <a href="https://jira.opendaylight.org/browse/CONTROLLER-1751" title="Sporadic cluster failure when member is restarted in OF cluster test" class="issue-link" data-issue-key="CONTROLLER-1751"><del>CONTROLLER-1751</del></a> where node connectivity occasionally gets delayed in the test environment for some reason (hypothesis is overload in the environment).</p> <p>I see in the bug summary that the test brings 2 out of the 3 nodes down then back up. The way akka works is that is that if a node becomes unreachable then it must become reachable or "downed" before it can allow another node to join. So if 2 nodes are unreachable then both have to come back before it allows either back in. We've seen previously that this can take more than 5 min due to suspected intermittent load in the environment with 2 nodes restarting (and whatever other load there is in the VM environment). So most of the time it's relatively quick but sometimes it can take arbitrarily longer. So I would suggest increasing the 5 min expectation to avoid intermittent timeouts (I'd say at least 15 min to be safe).</p> <p>I wonder if the tests should use more aggressive retry timings. Also are ODL2/ODL3 brought up concurrently?</p> <p>I meant retry timers at akka.conf layer &#8211; as I think they do back off by default.</p> <p>(In reply to Tom Pantelis from comment #3)<br/> &gt; I see in the bug summary that the test brings 2 out of the 3 nodes down then<br/> &gt; back up. The way akka works is that is that if a node becomes unreachable<br/> &gt; then it must become reachable or "downed" before it can allow another node<br/> &gt; to join. So if 2 nodes are unreachable then both have to come back before it<br/> &gt; allows either back in. We've seen previously that this can take more than 5<br/> &gt; min due to suspected intermittent load in the environment with 2 nodes<br/> &gt; restarting (and whatever other load there is in the VM environment). So most<br/> &gt; of the time it's relatively quick but sometimes it can take arbitrarily<br/> &gt; longer. So I would suggest increasing the 5 min expectation to avoid<br/> &gt; intermittent timeouts (I'd say at least 15 min to be safe).</p> <p>ok, I can change the test to wait for 15m for cluster sync, but what worries<br/> me is that later in the same job, there are more fundamental failures from<br/> netvirt's perspective (e.g. created networks not ending up in the config<br/> store). Because of that I'm worried that something is just broken under<br/> the hood and waiting might not be the answer.</p> <p>but, I'll try.</p> <p>(In reply to Robert Varga from comment #5)<br/> &gt; I meant retry timers at akka.conf layer &#8211; as I think they do back off by<br/> &gt; default.</p> <p>do you have an example I can look at? It's not totally clear<br/> to me what values I should tweak. This is worth a try too.</p> <p>(In reply to Robert Varga from comment #4)<br/> &gt; I wonder if the tests should use more aggressive retry timings. Also are<br/> &gt; ODL2/ODL3 brought up concurrently?</p> <p>yeah, ODL2/ODL3 are brought up in the same 1-2s window.</p> <p>just wanted to note that I have also seen this recently in a local setup with a recent Oxygen build</p> <p>On the kernel call, you mentioned you saw this stopping/restarting 1 node. This issue is almost a year old was related to 2 nodes stopping/restarting. From earlier analysis and comments from last year, it was determined that the 2 nodes did actually re-join but it happened after the 5 min test deadline. It was suggested to increases the test timeout. I assume there is still a controller test that does this (I know there was originally)? If so, if it hasn't been failing then can we close this issue?</p> <p>so we want to use a new bug then?</p> <p>Well we already have <a href="https://jira.opendaylight.org/browse/CONTROLLER-1849" title="controller not coming up healthy after being killed and restarted (401 after 5m)" class="issue-link" data-issue-key="CONTROLLER-1849"><del>CONTROLLER-1849</del></a> for a 1 node restart not rejoining. I assume the failure you saw that prompted your recent comment here was the same issue. So unless this issue is still current then let's close it.</p> <p>Keep in mind - the 401's due to the failed AAA reads and SyncStatus remaining false are all symptoms that the node did not rejoin the cluster on restart for whatever reason. At this point, AFAIK, this is only being seen occasionally when the first seed node is killed/restarted (ie <a href="https://jira.opendaylight.org/browse/CONTROLLER-1849" title="controller not coming up healthy after being killed and restarted (401 after 5m)" class="issue-link" data-issue-key="CONTROLLER-1849"><del>CONTROLLER-1849</del></a>).</p> <p>ok, I follow that logic. we'll open this back up if there is something specific about 2 nodes going down/up,<br/> and we have 1849 to track all that we think is left at this point.</p> <p>going with the assumption that this is not seen any more in this specific scenario<br/> where 2 of 3 nodes are going down and back up. We do have a very similar bug<br/> in <a href="https://jira.opendaylight.org/browse/CONTROLLER-1849" title="controller not coming up healthy after being killed and restarted (401 after 5m)" class="issue-link" data-issue-key="CONTROLLER-1849"><del>CONTROLLER-1849</del></a> that only deals with one node (first seed node) going down<br/> and up. We'll track that problem there.</p> Development External issue ID 9133 External issue URL Rank 0|i02slz: Status Whiteboard csit:sporadic_failures