https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/CONTROLLER-1488 controller <p>Problem: Our EntityOwnershipListener does not receive all callbacks. We have run several tests and we see unexpected behavior:</p> <p>1. Some cluster nodes do not receive callback after Controller cold start (this node basically does not know whether it is master or slave) (see log 11:57 at node1)</p> <p>2. Some cluster nodes do not receive callback (hasOwner=false) after Shard Cluster Leader Node has <br/> been isolated. This is problem because Leader can still think that it is Leader. (see log 12:25 at node2)</p> <p>This has impact Fail-over Cluster behavior of our custom application and OpenFlow Plugin.</p> <p>Enviroment:</p> <ul class="alternate" type="square"> <li>3 node cluster</li> <li>Ubuntu</li> <li>Beryllium (ZIP downloaded 2016-02-23)</li> <li>Modified existing test application (artifactId=clustering-it-provider groupId=org.opendaylight.controller)</li> </ul> <p>See attached karaf.log, configuration files and source codes.</p> <p>Operating System: All<br/> Platform: All</p> CONTROLLER-1488

EntityOwnershipListener missing ownershipChanged() callback

Bug Resolved Done Robert Varga Jozef Slezák Wed, 24 Feb 2016 14:59:52 +0000 Thu, 26 May 2016 12:37:24 +0000 Thu, 26 May 2016 12:37:24 +0000 Beryllium clustering 0 5 <p>It would help if you could outline detailed reproduction steps for what you're seeing.</p> <p>Atatchment contains following files<br/> 1/ OwnershipListener-logTimesAndEvents.xlsx<br/> Excel file containing small summary of events on 3 cluster nodes <br/> 2/ OwnershipListener-logSummary.log<br/> Extract logs from OwnershipListener test class implementation from all 3 cluster nodes (see marking from the beginning, with ref. to karaf file)<br/> 3/ Node1-..., 2, 3 subdirectories<br/> Complete karaf.logs from nodes + grepped OwnershipListener class events (ownl3 - used for 2/ creation )</p> <p>Attachment 24022016-ClusterOwnershipListener.zip has been added with description: Karaf logs from test and table sumarizing it</p> <p>Hi Tom,<br/> Thanks for quick response. Here comes more details about our tests executed.</p> <p>Environment:<br/> 3 Ubuntu VMs running on Oracle VM Virtual Machine (W7,64,16G RAM), ver. 5.0.14<br/> Each VM running:</p> <ul class="alternate" type="square"> <li>Ubuntu 14.04.2 LTS (3G RAM)</li> <li>Berilium distribution (fresh from <a href="https://www.opendaylight.org/software/downloads/beryllium" class="external-link" target="_blank" rel="nofollow noopener">https://www.opendaylight.org/software/downloads/beryllium</a>)</li> <li>extended test application (org.opendaylight.controller:clustering-it) by<br/> OwnershipListener class</li> </ul> <p>Steps done:<br/> 1/ Start 3 controllers<br/> 2/ install replaced odl-clustering-test-app<br/> 3/ wait until everything started, incl. odl-clustering-test-app<br/> 4/ run python ./isolate.py 1 (isolate leader)<br/> 5/ wait until everything stabilize<br/> 6/ rejoin 1<br/> 7/ Stop instances</p> <p>Used also python ./monitor.py to monitor cluster instances <br/> (PS during the test 2 instances had been restarted because of some problems)</p> <p>Adapted sample application sources based on<br/> <a href="https://github.com/opendaylight/controller/tree/stable/beryllium/opendaylight/md-sal/samples/clustering-test-app" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/opendaylight/controller/tree/stable/beryllium/opendaylight/md-sal/samples/clustering-test-app</a><br/> (older release adapted)</p> <p>Attachment clustering-ownership-test-app.zip has been added with description: Adapted sample application sources</p> <p>I'm not sure if I completely follow/understand the spreadsheet. From what I can gather, after startup, all 3 nodes registered a candidate and listener and all were notified of ownership state with node1 obtaining ownership of the entity. </p> <p>Wrt to "isolate", I assume you broke the network connection between node1 and the other 2 nodes causing a network partition. So node1 became isolated and node2 became the new EOS leader and was granted ownership. node 3's listener was also notified of the ownership change.</p> <p>However node1's listener did not receive a notification that it lost ownership. This is b/c it was an isolated leader and thus can no longer make progress, i.e. it can't commit ownership changes b/c it can't get a majority consensus.</p> <p>After the network partition was healed, node1 became a follower to node2 and synced up at which point node1's listener was notified of the ownership change(s).</p> <p>It seems this scenario corresponds to #2 in the bug desc.</p> <p>I'm unclear as to the rest of sequence in the spreadsheet.</p> <p>I'm not clear on what #1 means but it sounds like <a href="https://git.opendaylight.org/gerrit/#/c/34674/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/34674/</a>. Do you have that patch?</p> <p>Excel file corresponds to manual events and to EntityOwnershipListener notification events.</p> <p>Excel lines with cell content Leader/Slave corresponds to all kafaf log records corresponding to EntityOwnershipListener notification events. You can also see timestamp from karaf.log overthere.</p> <p>Log entries are from <br/> OwnershipListener.ownershipChanged(EntityOwnershipChange entityOwnershipChange)<br/> method (entry to it). Additionally you can see also listener registration and closing (Created instance, Close session events)</p> <p>I haven't applied patch extra, so it depends it is in prebuild Berillium<br/> Uploaded slightly modified excel.</p> <p>Attachment OwnershipListener-logTimesAndEvents<span class="error">&#91;1&#93;</span>.xlsx has been added with description: Updated excel file (comments)</p> <p>Tom, that sounds correct. It is true that the minority partition cannot make forward progress when touching internal ODL resources, but EOS is usually used to present a single master to outside resources.</p> <p>Since the connectivity affects only cluster nodes, an outside entity (like a OF switch) connected to both partitions will see two leaders.</p> <p>Hence if we know we are in the minority partition, we need to bring the master down. I am not sure if this should be done via a normal 'isOwner=false hasOwner=false' report, or via a more dire means.</p> <p>Yeah that's a hole resulting from using the dataTree/raft to trigger ownership change events. It seems the minority node would have to generate a "fake" event with "wasOwner=true, isOwner=false". However the hasOwner part wouldn't be known. It seems we would need "hasOwner=unknown" state.</p> <p>I think this may need another type of explicit event for this case. But we're limited with the current API. The new API that was put into mdsal in Be would give us the flexibility to do this. We need to implement the new API in Boron - another thing to put on the list <img class="emoticon" src="https://jira.opendaylight.org/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>In terms of knowing if we are in a minority partition may be tricky. If the EOS leader becomes partitioned, we detect that via the IsolatedLeader raft state. However it doesn't know if the majority of followers are actually down or if they're up and there's a network partition. Is it valid to send en event indicating the local node should no longer be master w/o knowing whether or not a majority cluster still remains that will elect a new leader and re-assign ownership?</p> <p>(In reply to Robert Varga from comment #8)<br/> &gt; Tom, that sounds correct. It is true that the minority partition cannot make<br/> &gt; forward progress when touching internal ODL resources, but EOS is usually<br/> &gt; used to present a single master to outside resources.<br/> &gt; <br/> &gt; Since the connectivity affects only cluster nodes, an outside entity (like a<br/> &gt; OF switch) connected to both partitions will see two leaders.<br/> &gt; <br/> &gt; Hence if we know we are in the minority partition, we need to bring the<br/> &gt; master down. I am not sure if this should be done via a normal<br/> &gt; 'isOwner=false hasOwner=false' report, or via a more dire means.</p> <p>Based on Robert's comment, it looks like we've identified #2.</p> <p>But I'm still not clear on #1 - need more explanation...</p> <p>(In reply to Tom Pantelis from comment #5)<br/> &gt; I'm not sure if I completely follow/understand the spreadsheet. From what I<br/> &gt; can gather, after startup, all 3 nodes registered a candidate and listener<br/> &gt; and all were notified of ownership state with node1 obtaining ownership of<br/> &gt; the entity. <br/> &gt; <br/> &gt; Wrt to "isolate", I assume you broke the network connection between node1<br/> &gt; and the other 2 nodes causing a network partition. So node1 became isolated<br/> &gt; and node2 became the new EOS leader and was granted ownership. node 3's<br/> &gt; listener was also notified of the ownership change.<br/> &gt; <br/> &gt; However node1's listener did not receive a notification that it lost<br/> &gt; ownership. This is b/c it was an isolated leader and thus can no longer make<br/> &gt; progress, i.e. it can't commit ownership changes b/c it can't get a majority<br/> &gt; consensus.<br/> &gt; <br/> &gt; After the network partition was healed, node1 became a follower to node2 and<br/> &gt; synced up at which point node1's listener was notified of the ownership<br/> &gt; change(s).<br/> &gt; <br/> &gt; It seems this scenario corresponds to #2 in the bug desc.<br/> &gt; <br/> &gt; I'm unclear as to the rest of sequence in the spreadsheet.<br/> &gt; <br/> &gt; I'm not clear on what #1 means but it sounds like<br/> &gt; <a href="https://git.opendaylight.org/gerrit/#/c/34674/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/34674/</a>. Do you have that patch?</p> <p>I'm having trouble following/understanding the spreadsheet, e.g. layout and terminology. It would be help if you could outline it step by step, i.e. what actions were taken, what occurred and what didn't occur that was expected, e.g.</p> <ul class="alternate" type="square"> <li>started all 3 nodes</li> <li>node2 got change event: wasOwner: F, isOwner: T, hasOwner: T (expected)</li> <li>node1 got change event: wasOwner: F, isOwner: F, hasOwner: T (expected)</li> <li>stopped the karaf process on node2</li> <li>node1 got change event: wasOwner: F, isOwner: T, hasOwner: T (expected) OR expected node1 change event: wasOwner: F, isOwner: T, hasOwner: T did not occur</li> </ul> <p>(In reply to Tom Pantelis from comment #10)<br/> &gt; Based on Robert's comment, it looks like we've identified #2.<br/> &gt; <br/> &gt; But I'm still not clear on #1 - need more explanation...<br/> &gt; <br/> &gt; (In reply to Tom Pantelis from comment #5)<br/> &gt; &gt; I'm not sure if I completely follow/understand the spreadsheet. From what I<br/> &gt; &gt; can gather, after startup, all 3 nodes registered a candidate and listener<br/> &gt; &gt; and all were notified of ownership state with node1 obtaining ownership of<br/> &gt; &gt; the entity. <br/> &gt; &gt; <br/> &gt; &gt; Wrt to "isolate", I assume you broke the network connection between node1<br/> &gt; &gt; and the other 2 nodes causing a network partition. So node1 became isolated<br/> &gt; &gt; and node2 became the new EOS leader and was granted ownership. node 3's<br/> &gt; &gt; listener was also notified of the ownership change.<br/> &gt; &gt; <br/> &gt; &gt; However node1's listener did not receive a notification that it lost<br/> &gt; &gt; ownership. This is b/c it was an isolated leader and thus can no longer make<br/> &gt; &gt; progress, i.e. it can't commit ownership changes b/c it can't get a majority<br/> &gt; &gt; consensus.<br/> &gt; &gt; <br/> &gt; &gt; After the network partition was healed, node1 became a follower to node2 and<br/> &gt; &gt; synced up at which point node1's listener was notified of the ownership<br/> &gt; &gt; change(s).<br/> &gt; &gt; <br/> &gt; &gt; It seems this scenario corresponds to #2 in the bug desc.<br/> &gt; &gt; <br/> &gt; &gt; I'm unclear as to the rest of sequence in the spreadsheet.<br/> &gt; &gt; <br/> &gt; &gt; I'm not clear on what #1 means but it sounds like<br/> &gt; &gt; <a href="https://git.opendaylight.org/gerrit/#/c/34674/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/34674/</a>. Do you have that patch?</p> <p>See Sheet Nr2 listed events in different form<br/> Contains main events from logs (INIT/CLOSE/LEADER/FOLLOW) for each node as they happened. In comment you can also see matching line on LEADER.</p> <p>( similar to sequence chart )<br/> From the excel you can see also problematic points 1/ and 2/<br/> Hope it helps better, sorry for confutions with original form (the same data)<img class="emoticon" src="https://jira.opendaylight.org/images/icons/emoticons/sad.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>Attachment OwnershipListener-logTimesAndEventsV2.xlsx has been added with description: Inserted page with time-sorted events from logs</p> <p>Thanks that helps a lot. Case 1 looks like you're missing <a href="https://git.opendaylight.org/gerrit/#/c/34674/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/34674/</a>. It was merged to stable/be on Feb 19th. I would suggest retrying case 1 with the latest stable/be to ensure you have that patch.</p> <p>(In reply to Tibor Sirovatka from comment #12)<br/> &gt; Created attachment 853 <span class="error">&#91;details&#93;</span><br/> &gt; Inserted page with time-sorted events from logs<br/> &gt; <br/> &gt; See Sheet Nr2 listed events in different form<br/> &gt; Contains main events from logs (INIT/CLOSE/LEADER/FOLLOW) for each node as<br/> &gt; they happened. In comment you can also see matching line on LEADER.<br/> &gt; <br/> &gt; ( similar to sequence chart )<br/> &gt; From the excel you can see also problematic points 1/ and 2/<br/> &gt; Hope it helps better, sorry for confutions with original form (the same<br/> &gt; data)<img class="emoticon" src="https://jira.opendaylight.org/images/icons/emoticons/sad.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>Can you give me a quick info, how I can check merge is in or not?<br/> ( I downloaded prepacked karaf on 23th in the morning)<br/> It would help. Thanks!</p> <p>(In reply to Tom Pantelis from comment #13)<br/> &gt; Thanks that helps a lot. Case 1 looks like you're missing<br/> &gt; <a href="https://git.opendaylight.org/gerrit/#/c/34674/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/34674/</a>. It was merged to stable/be<br/> &gt; on Feb 19th. I would suggest retrying case 1 with the latest stable/be to<br/> &gt; ensure you have that patch.<br/> &gt; <br/> &gt; (In reply to Tibor Sirovatka from comment #12)<br/> &gt; &gt; Created attachment 853 <span class="error">&#91;details&#93;</span><br/> &gt; &gt; Inserted page with time-sorted events from logs<br/> &gt; &gt; <br/> &gt; &gt; See Sheet Nr2 listed events in different form<br/> &gt; &gt; Contains main events from logs (INIT/CLOSE/LEADER/FOLLOW) for each node as<br/> &gt; &gt; they happened. In comment you can also see matching line on LEADER.<br/> &gt; &gt; <br/> &gt; &gt; ( similar to sequence chart )<br/> &gt; &gt; From the excel you can see also problematic points 1/ and 2/<br/> &gt; &gt; Hope it helps better, sorry for confutions with original form (the same<br/> &gt; &gt; data)<img class="emoticon" src="https://jira.opendaylight.org/images/icons/emoticons/sad.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>So when you tested you had obtained a build/distro on Feb 23rd... where did you get it from? Was it the Be release distro? If so then Roberts patch wouldn't be in there since it was merged port release.</p> <p>Yes, it was path from official announcement, so not there.<br/> ( <a href="https://www.opendaylight.org/software/downloads/beryllium" class="external-link" target="_blank" rel="nofollow noopener">https://www.opendaylight.org/software/downloads/beryllium</a> )</p> <p>(In reply to Tom Pantelis from comment #15)<br/> &gt; So when you tested you had obtained a build/distro on Feb 23rd... where did<br/> &gt; you get it from? Was it the Be release distro? If so then Roberts patch<br/> &gt; wouldn't be in there since it was merged port release.</p> <p>Tomorrow we are planing to retest using config-parent version=0.4.1-SNAPSHOT to see whether the patch helped. I will give you the update.</p> <p>retested and problem 1 seems to be solved, but problem 2 is still there</p> <p>added similar sorted events in excel file - events on 3node cluster and ownership state on nodes</p> <p>Attachment OwnershipListener-logTimesAndEventsV2_public.xlsx has been added with description: sorted events after retest</p> <p>The second part requires exposing cluster synchronization status to the application, so it can detect a split-brain situation and act accordingly.</p> <p>In our RAFT implementation (which backs Entity Ownership Service) this happens during elections (since we do not know who the leader is) and when the leader becomes isolated (since the peers which it cannot reach can perform new elections).</p> <p>Applications which need to be sure that the cluster is in consistent state for them to operate can choose to suspend execution (without actually stopping) while EOS remains in jeopardy.</p> <p>master: <a href="https://git.opendaylight.org/gerrit/36752" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/36752</a></p> <p>To b(In reply to Robert Varga from comment #19)<br/> &gt; The second part requires exposing cluster synchronization status to the<br/> &gt; application, so it can detect a split-brain situation and act accordingly.<br/> &gt;<br/> &gt; In our RAFT implementation (which backs Entity Ownership Service) this<br/> &gt; happens during elections (since we do not know who the leader is) and when<br/> &gt; the leader becomes isolated (since the peers which it cannot reach can<br/> &gt; perform new elections).<br/> &gt; <br/> &gt; Applications which need to be sure that the cluster is in consistent state<br/> &gt; for them to operate can choose to suspend execution (without actually<br/> &gt; stopping) while EOS remains in jeopardy.<br/> &gt;<br/> &gt; master: <a href="https://git.opendaylight.org/gerrit/36752" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/36752</a></p> <p>To be clear, you never really know when you're in split brain. You only know the last time you were able to make progress or hear about progress. You can decide that if no progress has been made for 30 seconds (or an hour or whatever) that you're in a split-brain mode, but you'll never know.</p> <p>Adding another mechanism to use/detect this seems dangerous.</p> <p>I do not disagree, but I think you misunderstand what the patch does, or I misunderstand the point you are making. Since you have made the comments after this has been thoroughly discussed on the Clustering Hackers' call, I would like you to be more explicit about your assumptions and analysis so that we can be sure we are on the same page.</p> <p>The patch certainly does not introduce another detection mechanism, but rather exposes the confidence of the <b>EOS implementation</b> in the value it is reporting.</p> <p>The confidence level is directly tied to RAFT (which is an implementation detail and the application has no notion of it existing), e.g. this triggers when we:<br/> a) do not have consensus based on AppendEntries<br/> b) are not sure that our local view represents cluster-wide consensus</p> <p>Attachment OwnershipListener-logTimesAndEvents-patch36752-05042016.xlsx has been added with description: Excel with summary from retest with patch applied</p> <p>Attachment 05042016-patchTest-karaf-logs.zip has been added with description: karaf.log files from retest with patch applied</p> <p>Retested with following artefacts:</p> <p>1/ base Berilium build (<a href="https://github.com/opendaylight/integration-distribution/tree/stable/beryllium/distribution-karaf" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/opendaylight/integration-distribution/tree/stable/beryllium/distribution-karaf</a>, 1.6.2-SNAPSHOT )<br/> 2/ applied patch files (<a href="https://git.opendaylight.org/gerrit/36752" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/36752</a>)<br/> 3/ build and replaced following artefacts in deployment<br/> &#8211; system/org/opendaylight/controller/sal-common-api/1.3.2-SNAPSHOT/sal-common-api-1.3.2-SNAPSHOT.jar<br/> &#8211; system/org/opendaylight/controller/sal-distributed-datastore/1.3.2-SNAPSHOT/sal-distributed-datastore-1.3.2-SNAPSHOT.jar<br/> 4/ updated odl-clustering-test-app and deployed in karaf<br/> 4/ set DEBUG info for clustering components<br/> log:set DEBUG org.opendaylight.controller.md.sal.common.api.clustering<br/> log:set DEBUG org.opendaylight.controller.cluster</p> <p>CASE 1 - OK (as already verified by Michal)<br/> CASE 2 - OK, After isolation, Leader is getting inJeopardy=true state<br/> and becomming IsolatedLeader (still isOwner=true); other 2 controllers are voting for new Leader, one of them becomming new Leader<br/> After rejoin, inJeopardy=false and new Leader is electing from all 3 controllers.</p> <p>So it seems this case should be handled correctly with inJeopardy state<br/> (signalling just Leader had been isolated)</p> <p>-----------------------------------------------------------------------------<br/> PS: I need to recheck sync status between cluster-monitor.py and notification event, because it seemed from other test they are not in sync.<br/> Reason can be Shard registered by listener</p> <p>2016-04-05 11:40:03,516 | DEBUG | lt-dispatcher-17 | <br/> EntityOwnershipShard | 166 - org.opendaylight.controller.sal-akka-<br/> raft - 1.3.2.SNAPSHOT | member-1-shard-entity-ownership-operational: onRegisterListenerLocal: RegisterListenerLocal <span class="error">&#91;entityType=node-cluster-shard-info, listener=org.opendaylight.controller.ownership.OwnershipListener@18327f39&#93;</span></p> <p>In case of any problem found will issue separate ticket for it.</p> <p>(In reply to Tibor Sirovatka from comment #24)<br/> &gt; Retested with following artefacts:<br/> &gt; <br/> &gt; 1/ base Berilium build<br/> &gt; (<a href="https://github.com/opendaylight/integration-distribution/tree/stable/" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/opendaylight/integration-distribution/tree/stable/</a><br/> &gt; beryllium/distribution-karaf, 1.6.2-SNAPSHOT )<br/> &gt; 2/ applied patch files (<a href="https://git.opendaylight.org/gerrit/36752" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/36752</a>)<br/> &gt; 3/ build and replaced following artefacts in deployment<br/> &gt; &#8211;<br/> &gt; system/org/opendaylight/controller/sal-common-api/1.3.2-SNAPSHOT/sal-common-<br/> &gt; api-1.3.2-SNAPSHOT.jar<br/> &gt; &#8211;<br/> &gt; system/org/opendaylight/controller/sal-distributed-datastore/1.3.2-SNAPSHOT/<br/> &gt; sal-distributed-datastore-1.3.2-SNAPSHOT.jar<br/> &gt; 4/ updated odl-clustering-test-app and deployed in karaf<br/> &gt; 4/ set DEBUG info for clustering components<br/> &gt; log:set DEBUG org.opendaylight.controller.md.sal.common.api.clustering<br/> &gt; log:set DEBUG org.opendaylight.controller.cluster<br/> &gt; <br/> &gt; CASE 1 - OK (as already verified by Michal)<br/> &gt; CASE 2 - OK, After isolation, Leader is getting inJeopardy=true state<br/> &gt; and becomming IsolatedLeader (still isOwner=true); other 2 controllers are<br/> &gt; voting for new Leader, one of them becomming new Leader<br/> &gt; After rejoin, inJeopardy=false and new Leader is electing from all 3<br/> &gt; controllers.<br/> &gt; <br/> &gt; So it seems this case should be handled correctly with inJeopardy state<br/> &gt; (signalling just Leader had been isolated)<br/> &gt; <br/> &gt; <br/> &gt; -----------------------------------------------------------------------------<br/> &gt; PS: I need to recheck sync status between cluster-monitor.py and<br/> &gt; notification event, because it seemed from other test they are not in sync.<br/> &gt; Reason can be Shard registered by listener<br/> &gt; <br/> &gt; 2016-04-05 11:40:03,516 | DEBUG | lt-dispatcher-17 | <br/> &gt; EntityOwnershipShard | 166 -<br/> &gt; org.opendaylight.controller.sal-akka-<br/> &gt; raft - 1.3.2.SNAPSHOT | member-1-shard-entity-ownership-operational:<br/> &gt; onRegisterListenerLocal: RegisterListenerLocal<br/> &gt; [entityType=node-cluster-shard-info,<br/> &gt; listener=org.opendaylight.controller.ownership.OwnershipListener@18327f39]<br/> &gt; <br/> &gt; In case of any problem found will issue separate ticket for it.</p> <p>Leader status is correct (overlooked another leader change and following notification 5 minutes later). Anyway member-1-shard-entity-ownership-operational is not listed in modules.conf cluster configuration file.</p> <p>That is by design. This shard is dynamically created on startup as it must exist on all nodes.</p> <p>(In reply to Tibor Sirovatka from comment #25)<br/> &gt; member-1-shard-entity-ownership-operational is not listed in modules.conf<br/> &gt; cluster configuration file.</p> <p>Thanks Tom for quick answer ! How can I specify, I am interesting in Leader changes in specific application. (We want to use OwnershipListener as general service for our modules)<br/> Thanks a lot!<br/> (In reply to Tom Pantelis from comment #26)<br/> &gt; That is by design. This shard is dynamically created on startup as it must<br/> &gt; exist on all nodes.<br/> &gt; <br/> &gt; (In reply to Tibor Sirovatka from comment #25)<br/> &gt; &gt; member-1-shard-entity-ownership-operational is not listed in modules.conf<br/> &gt; &gt; cluster configuration file.</p> <p>(In reply to Robert Varga from comment #19)<br/> &gt; The second part requires exposing cluster synchronization status to the<br/> &gt; application, so it can detect a split-brain situation and act accordingly.<br/> &gt; <br/> &gt; In our RAFT implementation (which backs Entity Ownership Service) this<br/> &gt; happens during elections (since we do not know who the leader is) and when<br/> &gt; the leader becomes isolated (since the peers which it cannot reach can<br/> &gt; perform new elections).<br/> &gt; <br/> &gt; Applications which need to be sure that the cluster is in consistent state<br/> &gt; for them to operate can choose to suspend execution (without actually<br/> &gt; stopping) while EOS remains in jeopardy.<br/> &gt; <br/> &gt; master: <a href="https://git.opendaylight.org/gerrit/36752" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/36752</a></p> <p>As an aside, it seems like "provide hook to detect split brain in entity ownership service" should really be a separate bug from this one.</p> Blocks CONTROLLER-1492 MDSAL-158 Development External issue ID 5414 External issue URL Issue Type ODL SR Target Milestone Rank 0|i02qvr: