https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/CONTROLLER-1865 controller <p><a href="https://lists.opendaylight.org/pipermail/controller-dev/2018-September/014624.html" class="external-link" target="_blank" rel="nofollow noopener">initial email conversation </a></p> <p>CSIT is sporadically failing when finding that basic-rpc-test ownership has changed from<br/> it's current owner to some new owner after bringing back an isolated node. The expectation<br/> is that no ownership should change when a new node is re-joined to the cluster.</p> <p><a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/445/jamo-controller-csit-3node-clustering-ask-all-oxygen/1/robot-plugin/log.html.gz#s1-s14-t9-k2-k1-k1-k4" class="external-link" target="_blank" rel="nofollow noopener">an example of this failure </a></p> <p>the above example was run with debugs enabled for</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> controller.cluster.datastore.entityownership:DEBUG </pre> </div></div> <p>Note the reproduction above ran this test suite 8 times, but it only failed once.</p> <p><a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/445/jamo-controller-csit-3node-clustering-ask-all-oxygen/1/" class="external-link" target="_blank" rel="nofollow noopener">Karaf logs here </a></p> <p>In order to extract just the portion of logs from the single test suite that failed, you can run<br/> this command:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> sed -n <span class="code-quote">"/16.59.*ROBOT MESSAGE: Starting suite/,/restart_odl_with_tell_based_false/p"</span> odl1_karaf.log </pre> </div></div> <p>the main piece to that command is the 16.59 which is taken from a timestamp I found<br/> that will be seen in all of the karaf logs for that specific suite's start point.</p> CONTROLLER-1865

owner changed after bringing back isolated node

Bug Medium Resolved Done Tom Pantelis Jamo Luhrsen csit:3node Tue, 2 Oct 2018 17:36:50 +0000 Sat, 22 Dec 2018 15:45:24 +0000 Sat, 22 Dec 2018 15:45:24 +0000 Neon Oxygen SR4 Fluorine SR2 clustering 0 5 <p>AFAIR this has been always the behavior, it seems like when isolated instance rejoins it forces an owner re-election. Long time back I commented a test in OFP checking this because I was told this behavior was expected.</p> <p>The test isolates the node that originally owns the Basic-rpc-test entity. In this run that was odl3. During isolation, the majority partition, odl1 and old2, elects a new owner since it is able to make progress. In this case that was odl2. When odl3 was un-isolated, since it re-joins as a minority member, it should not depose the new owner in the majority partition, ie the owner should remain odl2. This part actually worked as designed. The problem is that on partition heal, the new EOS leader, odl1, received a notification from akka that odl2 was unreachable:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>2018-10-02T17:01:16,076 | INFO | opendaylight-cluster-data-shard-dispatcher-76 | ShardManager | 287 - org.opendaylight.controller.sal-distributed-datastore - 1.7.4.SNAPSHOT | Received UnreachableMember: memberName MemberName{name=member-2}, address: akka.tcp://opendaylight-cluster-data@10.30.170.39:2550 </pre> </div></div> <p>It then received an ReachableMember event about a second later at 17:01:17,557. But this little blip caused the EOS to re-assign ownership to odl3 since it had received a ReachableMember event for odl3 shortly before. </p> <p>The weird thing is that there's no "Marking node(s) as UNREACHABLE ..." log message from akka for odl2, just one for odl3 earlier which is expected since it was isolated. I don't know why akka sent a spurious UnreachableMember event for odl2 - I've never seen that happen w/o a corresponding UNREACHABLE log message. </p> <p>This was also reported in <a href="https://bugs.opendaylight.org/show_bug.cgi?id=8430" class="external-link" target="_blank" rel="nofollow noopener">https://bugs.opendaylight.org/show_bug.cgi?id=8430</a>.</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=tpantelis" class="user-hover" rel="tpantelis">tpantelis</a>, what do we want to do next with this one? I can try to reproduce it locally if<br/> that's helpful. Or we can run our jobs with some tweaks or different logging levels.</p> <p>Easiest thing to do short term is to not fail the test if the owner changes - functionally it doesn't really matter if it does. I can also try to put in a workaround to check the cluster status to verify if the member is really unreachable in the hopes that the spurious/false event doesn't actually reflect the real status (since there was no akka log message indicating unreachability). Other than that, we can try akka artery and hope that alleviates the spurious events.</p> <p>well, thinking worst case here and whatever causes this spurious owner change starts happening<br/> frequently, aren't we in a less-than optimal state? <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=vpickard" class="user-hover" rel="vpickard">vpickard</a>, didn't you have some environment<br/> that was seeing unexpected leader changes in your cluster? Was that strictly related to performance/scale<br/> testing?</p> <p>Obviously something is funky here and akka is default and what companies are deploying with. I understand<br/> it's not a big deal that once in a while Leadership changes, but this is a pretty simple test with<br/> pretty simple expectations. Hopefully we can find a way to explain it without just ignoring it.</p> <p>As to trying with artery, I had something working a while back to make testing in our infra possible, but<br/> I'll have to refresh my memory for what I did.</p> <p>Jamo,<br/> Yes, I'm currently running some scale testing downstream, and we have seen leadership changes in the cluster when none of the nodes were taken down.</p> <p>This is likely related to resource issues. I was monitoring the akka port (2550) on all cluster nodes, and it was clear that the recvq for the akka connection was not processed for an extended period of time (40+ seconds), which resulted in leadership changes. Analysis indicated that this was related to very long GC pauses (27+ seconds, 3 times in 20-30 min period). We had a JFR running, and this bug was opened (and patched) to help address the huge allocations. </p> <p>Still running scale testing with this patch <span class="error">&#91;1&#93;</span> for this JIRA <span class="error">&#91;2&#93;</span> (and also doing GC tunings) to see if we can get this more stable.</p> <p>When you see this failure in the CSIT, are you monitoring the akka ports on all cluster nodes?</p> <p>I'm attaching the simple script I use to monitor this (akkaMon.sh)</p> <p><span class="error">&#91;1&#93;</span> <a href="https://git.opendaylight.org/gerrit/#/c/78377/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/78377/</a><br/> <span class="error">&#91;2&#93;</span> <a href="https://jira.opendaylight.org/browse/OPNFLWPLUG-1047" class="external-link" rel="nofollow">https://jira.opendaylight.org/browse/OPNFLWPLUG-1047</a></p> <p>That is related to shard leadership changes which isn't tied to akka membership events so wouldn't be affected by spurious UnreachableMember events. This CSIT is related entity ownership which does use UnreachableMember events to possibly re-assign ownership. </p> <p>I'll push a patch to dump the akka cluster state on UnreachableMember event so we can see if that status is actually reflected in the backend state. If it isn't then we can workaround it and ignore the spurious event. Otherwise we can open a case with akka and hope they'll provide support but you know the first thing they will say is to try artery <img class="emoticon" src="https://jira.opendaylight.org/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>I tried dumping the cluster state on UnreachableMember event in my local env. I brought a node down and saw the expected "Marking node(s) as UNREACHABLE ..." log message just prior to the ShardManager receiving the UnreachableMember event. I expected to see the node reflected in the unreachable set but it wasn't <img class="emoticon" src="https://jira.opendaylight.org/images/icons/emoticons/sad.png" height="16" width="16" align="absmiddle" alt="" border="0"/> Of course when I checked JMX afterwards it was in the Unreachable set. So unfortunately it appears it's reflected in the "queriable" state some time after we receive the event. Maybe if we wait a little time or poll the state for a time to verify... but that would start to getting into the "ugly" territory.</p> <p>I say we try artery to see if it reproduces there. <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=jluhrsen" class="user-hover" rel="jluhrsen">jluhrsen</a> I cherry-picked your Oxygen patch to master:<br/> <a href="https://git.opendaylight.org/gerrit/#/c/78629/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/78629/</a>. Can you run the "Global Rpc Isolate" over and over with that patch?</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=tpantelis" class="user-hover" rel="tpantelis">tpantelis</a>, yeah I'll work on this. I have a csit patch that we need to account<br/> for a few differences in the environment when we use artery. I need to double check<br/> the actual cluster config deployment too.</p> <p>working on it here:<br/> <a href="https://jenkins.opendaylight.org/sandbox/job/jamo-controller-csit-3node-clustering-ask-all-neon/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/sandbox/job/jamo-controller-csit-3node-clustering-ask-all-neon/</a></p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=tpantelis" class="user-hover" rel="tpantelis">tpantelis</a>, I'm having trouble... the controllers are not coming up properly in the csit. I can<br/> tell the distro is the one coming from your neon cherry pick of my patch because the akka.conf<br/> file (you can see it in the console log) has the aeron-udp tweak. Next, I'm removing that .tcp<br/> string from akka.tcp in akka.conf. That's all I remember having to do before and I got it working<br/> (both locally and in csit).</p> <p>the controllers are giving a 401 Unauthorized to a GET on /restconf/modules</p> <p>can you think of anything?</p> <p>We need to look at the karaf log.</p> <p>I got it figured out <b>finally</b></p> <p><a href="https://jenkins.opendaylight.org/sandbox/job/jamo-controller-csit-3node-clustering-ask-all-neon/15/" class="external-link" target="_blank" rel="nofollow noopener">this job </a> ran and passed that global rpc isolate once. I'll start running this in<br/> loops to see if it reproduces.</p> <p>This bug also happens with artery. logs here:</p> <p><a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/566/jamo-controller-csit-3node-clustering-ask-all-neon/19/" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/566/jamo-controller-csit-3node-clustering-ask-all-neon/19/</a></p> <p>what's next <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=tpantelis" class="user-hover" rel="tpantelis">tpantelis</a></p> <p>It's an issue in akka so, assuming we want to pursue it, then the next step would be to enable akka cluster and remoting debug and open an issue in akka upstream. </p> <p>I think I have an idea what's happening. Since there's no "Marking node(s) as UNREACHABLE ..." log message, as is usually seen when the UnreachableMember event is issued, and the spurious event occurs when the partition is healed, I think that indicates that it wasn't the local node that observed the unreachability of the third node (it really shouldn't have) - it was the state of the third node as observed by the previously isolated node. So when gossip convergence occurs when the partition is healed, the previously isolated node reports that the third node is still unreachable from its vantage point. If so then technically the event may be valid even though it's from the vantage point of the previously isolated node. During isolation, split-brain occurs and, after isolation, both sides eventually converge on the correct state but, during that time, we may observe the temporary intermediate state. The fact that it's sporadic seems to bear this out, ie sometimes the previously isolated node may have already re-connected to the third node and reports a reachable state on convergence and thus we don't receive the spurious UnreachableMember event.</p> <p>If the UnreachableMember event included which node observed the unreachability then we could possibly filter it out but unfortunately it doesn't. Also, from my testing, querying the current cluster state when the UnreachableMember event is received unfortunately doesn't report the node as unreachable so we can't rely on that.</p> <p>The only recourse I see short term is to remove the expectation that the owner won't change after partition heal. If it does sometimes, while not ideal, functionally we still end up in a steady state with one owner in the cluster.</p> <p>For longer term, we could report it upstream and see what the akka folks say.</p> <p>PS - based on <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=rovarga" class="user-hover" rel="rovarga">rovarga</a> comment in <a href="https://bugs.opendaylight.org/show_bug.cgi?id=8430#c9" class="external-link" target="_blank" rel="nofollow noopener">https://bugs.opendaylight.org/show_bug.cgi?id=8430#c9</a>, I think he may have theorized the same or similar.</p> <p>ok, I think I can follow what you are saying. Essentially, when the isolated node is rejoining<br/> the cluster and they are "comparing notes", it might happen that it's not yet realized that it<br/> can connect to one of the nodes so reports it as unreachable and in that case we could get a new<br/> leader election? Does this mean that all nodes are up for grabs to become Leader when this <br/> happens? meaning the isolated node could also end up as the Leader?</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=tpantelis" class="user-hover" rel="tpantelis">tpantelis</a>, I think we could learn something if you reported this to the akka folks, even if it's to learn<br/> that this possibility is expected.</p> <p>as for what to do with the test, we may just have to ignore this one if it's going to<br/> randomly fail on us for a legit reason. But, I'm also wondering if there is a slightly<br/> more careful way to isolate and bring it back that would ensure leadership remains, but<br/> I can't really think of anything.</p> <p>&gt; Does this mean that all nodes are up for grabs to become Leader when this <br/> happens? meaning the isolated node could also end up as the Leader?</p> <p>Any node that is known to be reachable. The isolated node is reachable at that point so could become owner, in fact that does happen in some runs.</p> <p>I can't see any way of ensuring ownership remains from the CSIT perspective. It's dependent on network communication and the timing of when nodes re-connect and when gossip messages are received etc - no way to control that.</p> <p>curious, what's the flow and state of things if we first isolate a node (I)<br/> (doing it by blocking traffic <b>outgoing</b> on the isolated node) then only<br/> bringing back it's communication to the Leader (L) and not the healthy<br/> Follower (F)?</p> <p>Essentially, the Leader has two nodes it can talk with? Would the previously<br/> isolated node tell the Leader that it can't reach F and that causes a new<br/> Leader election? Or does the Leader keep the existing 2 node cluster with<br/> L and F?</p> <p>Not sure I follow exactly... what is L the leader of?</p> <p>I assume you mean the new Leader of the EOS shard that is elected in the majority partition after isolation. Let's say <b>L</b> and <b>F</b> are in the majority partition and <b>I</b> is isolated. So <b>L</b> and <b>F</b> can talk to each other but neither can talk to <b>I</b>. Let's say entity <b>E</b> is now owned by <b>F</b> in the majority partition. <b>I</b> also still owns <b>E</b> on its side b/c it doesn't know it's isolated. At this point we have 2 owners in the cluster. Then communication between <b>L</b> and <b>I</b> is re-established. Initially <b>I</b> rejoins and relinquishes ownership of <b>E</b> leaving <b>F</b> as the sole owner. So far so good. Via akka's gossip convergence, <b>L</b> learns from <b>I</b> that <b>F</b> is unreachable from its vantage point. So <b>L</b> receives an UnreachableMember event that <b>F</b> is unreachable and selects a new owner for <b>E</b>, which could be either <b>L</b> or <b>I</b>.</p> <p>Ok, I'm starting to understand better. can we try it the other way now?</p> <p> </p> <p><b>L</b> and <b>F</b> can talk to each other but not <b>I</b>, so like you said <b>I</b> owns<br/> <b>E</b> just like <b>L</b> owns <b>E</b> because it doesn't know it's isolated. Now,<br/> let <b>I</b> get communication to <b>F</b> instead of <b>L</b> this time.</p> <p>what is our end state then?</p> <p>That's an interesting scenario. If <b>I</b> was the shard leader before isolation then it would remain as IsolatedLeader during isolation. When connection is re-established to <b>F</b>, it would send AppendEntries but <b>F</b> would reject it b/c <b>L</b>'s term would be less than <b>F</b>'s (as <b>F</b>'s term was bumped due to the new leader election). <b>L</b> would change to Follower due to the AppendEntriesReply with a higher term. Eventually it would timeout due to not hearing from the leader <b>L</b> and start a new election. It would request a vote from <b>F</b> but would (or should) be rejected b/c <b>F</b>'s last log index and term would be higher (part of the brilliance of the Raft algorithm). <b>I</b> would then go back to Follower and repeat. So <b>I</b> would essentially remain isolated. </p> <p><b>L</b> would or should not get an UnreachableMember event for itself - that wouldn't make sense. Looking at the logs in this case, <b>F</b> (odl2) did not report an UnreachableMember event for itself.</p> <p>In any case, <b>L</b> would remain the owner of <b>E</b>.</p> <p>Looking at the logs again, odl1 reported the UnreachableMember event for odl2 at 17:01:16,076. odl3 reported the ReachableMember event for odl2 at 17:01:16,988 which is slightly after (912 ms) the UnreachableMember event. So that supports the theory. odl3 reported odl1 reachable at 17:01:16,001 and odl1 reported odl3 reachable at 17:01:15,732. So odl1 and odl3 would have started gossip convergence before odl3 gained reachability with odl2.</p> <p> </p> <p>Correct, the gory details are at <a href="https://doc.akka.io/docs/akka/2.5/common/cluster.html" class="external-link" target="_blank" rel="nofollow noopener">https://doc.akka.io/docs/akka/2.5/common/cluster.html</a></p> <p> </p> <p>yeah - given akka's gossip convergence, it's FAD (and not to say it's wrong either).</p> <p>thanks for walking me through these <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=tpantelis" class="user-hover" rel="tpantelis">tpantelis</a>. I'm learning a lot <img class="emoticon" src="https://jira.opendaylight.org/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>Since we cannot completely guarantee that this test case with an isolated node will keep the<br/> same Leader when it rejoined, I will remove that test case. The only other option I could think<br/> of would be to enhance the test case to get really "white box" and check the logs for<br/> those UnreachableMember messages and somehow figure out if the Leadership change<br/> (when it happens) is expected. But, now I think we are falling in to the land of testing akka<br/> which we just have to trust is already done by the akka project themselves.</p> <p>yeah - we could do something fancy like that, ie before failing the test due to owner change, check the EOS shard leader's log for ''Received UnreachableMember" for whichever the owner's member name was expected not to change. I don't think we'd  actually be testing akka but rather working around it so at least when the spurious event doesn't occur then we can verify no owner change. It's up to you if you think it's worth it - I'm fine either way.</p> Development Rank 0|i03j73: