OpenDaylight JIRA

OpenDaylight JIRA https://jira.opendaylight.org This file is an XML representation of an issue en-us

8.20.10

820010

22-06-2022

[MDSAL-291] Member isolation test fails in Carbon and Nitrogen https://jira.opendaylight.org/browse/MDSAL-291 mdsal <p>It works in Boron but not in Carbon or Nitrogen. New test will be added to catch this.</p> <p>Operating System: All<br/> Platform: All</p> MDSAL-291

Member isolation test fails in Carbon and Nitrogen

Bug Resolved Done Robert Varga Luis Gomez Tue, 12 Sep 2017 04:40:59 +0000 Fri, 9 Mar 2018 18:00:23 +0000 Wed, 18 Oct 2017 18:59:59 +0000 0 5 <p>Isnt this related to new cluster failures introduced possibly by this change? <a href="https://git.opendaylight.org/gerrit/#/c/62708/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/62708/</a></p> <p>Yes, patch date match the regression.</p> <p>Test added to track this: <a href="https://git.opendaylight.org/gerrit/#/c/63014" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/63014</a></p> <p>OK, current nitrogen branch shows this bug:</p> <p><a href="https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-3node-gate-clustering-only-nitrogen/280/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-3node-gate-clustering-only-nitrogen/280/</a></p> <p>However after applying this revert:</p> <p><a href="https://git.opendaylight.org/gerrit/#/c/63111/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/63111/</a></p> <p>The bug is gone:</p> <p><a href="https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-3node-gate-clustering-only-nitrogen/279" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-3node-gate-clustering-only-nitrogen/279</a></p> <p>So I think it makes sense to mark this regression as BLOCKER and move the issue to mdsal.</p> <p>To reproduce:</p> <p>Run OF cluster member isolation test:</p> <ul class="alternate" type="square"> <li>Connect switch to 3 instances</li> <li>isolate (e.g. using iptables) switch master instance from the cluster, wait for cluster to elect new master</li> <li>rejoin master instance, wait for cluster to be in sync</li> <li>isolate switch candidate instance from the cluster, wait for cluster to be in sync</li> <li>rejoin candidate instance, wait for cluster to be in sync</li> <li>Disconnect switch</li> </ul> <p>After this test, if we reconnect the same switch, the entity owner does not show correct number of connected instances + no instance is given mastership for the device, so everything fails for the device. To recover we have to restart controller.</p> <p>From these two lines:<br/> 2017-09-13 19:06:17,923 | INFO | lt-dispatcher-22 | ContextChainHolderImpl | 281 - org.opendaylight.openflowplugin.impl - 0.5.0.SNAPSHOT | Removing device from operational DS Uri <span class="error">[_value=openflow:1]</span> was successful<br/> 2017-09-13 19:06:23,103 | INFO | entLoopGroup-7-1 | ConnectionAdapterImpl | 287 - org.opendaylight.openflowplugin.openflowjava.openflow-protocol-impl - 0.5.0.SNAPSHOT | Hello received</p> <p>only the first one appears (with different time) in karaf.log <span class="error">[5]</span> of the failed run.<br/> I started running tests with debug logs for org.opendaylight.mdsal.singleton, but Sandbox had problems launching VMs for a while. So only one relevant run <span class="error">[6]</span> finished yet, but the test case passed.</p> <p><span class="error">[0]</span> > <a href="https://git.opendaylight.org/gerrit/#/c/62708/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/62708/</a></p> <p>One behavioral change is that service un-registration is now asynchronous, which (from user point of view) looks like subsequent re-registration is delayed, so perhaps the Robot check now hits this window of original master not being a candidate yet? Will need more time to investigate logs.</p> <p><span class="error">[5]</span> <a href="https://logs.opendaylight.org/releng/jenkins092/openflowplugin-csit-3node-gate-clustering-only-nitrogen/280/odl1_karaf.log.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/openflowplugin-csit-3node-gate-clustering-only-nitrogen/280/odl1_karaf.log.gz</a><br/> <span class="error">[6]</span> <a href="https://jenkins.opendaylight.org/sandbox/job/openflowplugin-csit-3node-clustering-only-nitrogen/4/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/sandbox/job/openflowplugin-csit-3node-clustering-only-nitrogen/4/</a></p> <p>If you see the test trend:</p> <p><a href="https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-3node-clustering-only-nitrogen/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-3node-clustering-only-nitrogen/</a></p> <p>It is not failing 100% of times but definitely there are more times it fails than it does not, and this started to happen just after the patch I reverted.</p> <p>Also I can easily reproduce the issue in my env.</p> <p>Also to be fair the member isolation test never worked 100% OK, so the real regression is that now it fails more often than before. From this perspective if the debug/fix is complicated at this time I think we can release nitrogen and write something in the release notes saying member isolation is not supported and therefore requires cluster restart.</p> <p>Theory is that the rework of <a href="https://jira.opendaylight.org/browse/MDSAL-275" title="VerifyException from flapping service singleton registration" class="issue-link" data-issue-key="MDSAL-275"><del>MDSAL-275</del></a> is flushing synchronization issues in OFP. I will review the logs tomorrow, but in the mean time it would be good to check ofp if it's solid.</p> <p>Previously singleton service would block the calling thread until after the unregistration has completely finished. This is no longer the case, as it does not make sense and it actually was unstable in face of GCs and other timing issues.</p> <p>Hence it is possible that the absence of the lock is flushing lack of synchronization downstream...</p> <p>Debug logs with the revert are here <span class="error">[8]</span> (until Friday ends).</p> <p>I have not reproduced a clear-cut replication of this symptom (without the revert) yet. Can someone with Sandbox rights (Luis?) keep hitting Rebuild on run #13 until the result looks right?<br/> (And also archive karaf logs somewhere.)</p> <p><span class="error">[8]</span> <a href="https://logs.opendaylight.org/sandbox/jenkins091/openflowplugin-csit-3node-clustering-only-nitrogen/11/odl1_karaf.log.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/sandbox/jenkins091/openflowplugin-csit-3node-clustering-only-nitrogen/11/odl1_karaf.log.gz</a></p> <p>> run #13</p> <p>Of job <span class="error">[9]</span>.</p> <p><span class="error">[9]</span> <a href="https://jenkins.opendaylight.org/sandbox/job/openflowplugin-csit-3node-clustering-only-nitrogen/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/sandbox/job/openflowplugin-csit-3node-clustering-only-nitrogen/</a></p> <p>I triggered a few, this one for instance is an example of the failure:</p> <p><a href="https://logs.opendaylight.org/sandbox/jenkins091/openflowplugin-csit-3node-clustering-only-nitrogen/17/odl1_karaf.log.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/sandbox/jenkins091/openflowplugin-csit-3node-clustering-only-nitrogen/17/odl1_karaf.log.gz</a></p> <p>You can see in the robot log, the first failure is already incorrect number or instances returned by entity owner.</p> <p>OK, I see things are cleared now in sandbox. Vratko, fyi I have updated your patch to disable switch ECHO requests as otherwise you can get undesired switch reconnects due to controller not responding to these messages.</p> <p>> Of job <span class="error">[9]</span>.</p> <p>This week, I got the desired outcome (only 1 failed test case) twice (runs #2 and #6), but both times there are exceptions which were not reported on releng.</p> <p>Cause:<br/> java.lang.IllegalStateException: Transaction ReadWriteShardDataTreeTransaction{id=member-1-datastore-operational-fe-0-chn-4-txn-13-0, closed=false, snapshot=MutableDataTree [modification=NodeModification <span class="error">[identifier=(urn:ietf:params:xml:ns:netconf:base:1.0)data, modificationType=NONE, childModification={}]</span>]} is open</p> <p>Consequence:<br/> 2017-09-18 14:59:32,592 | WARN | ofppool-0 | ClusterSingletonServiceGroupImpl | 262 - org.opendaylight.mdsal.singleton-dom-impl - 2.3.0.SNAPSHOT | Service group openflow:1 service stopping reported error<br/> java.util.concurrent.CancellationException: Task was cancelled.</p> <p>Other than that, the simplified job passed once (#3), and twice (#4, #5) it had additional failures in Cluster Sync Problems suite. Still waiting for a releng-compatible failure.</p> <p>Can we downgrade severity based on the information we have collected so far? Thanks!</p> <p>I am OK downgrading priority, what about you Vratko?</p> <p>Based on the logs provided by Luis, there seems to be an issue with the state machine when placeholders are in play:</p> <p>2017-09-15 00:20:41,415 | DEBUG | lt-dispatcher-21 | ClusterSingletonServiceGroupImpl | 262 - org.opendaylight.mdsal.singleton-dom-impl - 2.3.0.SNAPSHOT | Service group openflow:1 services started<br/> 2017-09-15 00:20:41,415 | DEBUG | lt-dispatcher-21 | ClusterSingletonServiceGroupImpl | 262 - org.opendaylight.mdsal.singleton-dom-impl - 2.3.0.SNAPSHOT | Service group openflow:1 switching from TAKING_OWNERSHIP to OWNER<br/> 2017-09-15 00:21:04,286 | DEBUG | entLoopGroup-5-5 | sterSingletonServiceProviderImpl | 262 - org.opendaylight.mdsal.singleton-dom-impl - 2.3.0.SNAPSHOT | Closing service group openflow:1<br/> 2017-09-15 00:21:04,286 | DEBUG | entLoopGroup-5-5 | ClusterSingletonServiceGroupImpl | 262 - org.opendaylight.mdsal.singleton-dom-impl - 2.3.0.SNAPSHOT | Service group openflow:1 unregistering<br/> 2017-09-15 00:21:04,286 | DEBUG | entLoopGroup-5-5 | ClusterSingletonServiceGroupImpl | 262 - org.opendaylight.mdsal.singleton-dom-impl - 2.3.0.SNAPSHOT | Service group openflow:1 closing<br/> 2017-09-15 00:21:04,286 | DEBUG | entLoopGroup-5-5 | sterSingletonServiceProviderImpl | 262 - org.opendaylight.mdsal.singleton-dom-impl - 2.3.0.SNAPSHOT | Replaced group openflow:1 with PlaceholderGroup</p> {id=openflow:1} <p>2017-09-15 00:21:04,308 | DEBUG | entLoopGroup-5-5 | sterSingletonServiceProviderImpl | 262 - org.opendaylight.mdsal.singleton-dom-impl - 2.3.0.SNAPSHOT | Call registrationService org.opendaylight.openflowplugin.impl.lifecycle.ContextCha<br/> inImpl@73da35a6 method for ClusterSingletonService Provider org.opendaylight.mdsal.singleton.dom.impl.DOMClusterSingletonServiceProviderImpl@215ba1af<br/> 2017-09-15 00:21:14,895 | DEBUG | lt-dispatcher-24 | sterSingletonServiceProviderImpl | 262 - org.opendaylight.mdsal.singleton-dom-impl - 2.3.0.SNAPSHOT | Ownership change for ClusterSingletonService Provider DOMEntityOwnershipChange [en<br/> tity=DOMEntity [type=org.opendaylight.mdsal.ServiceEntityType, id=/(urn:opendaylight:params:xml:ns:yang:mdsal:core:general-entity?revision=2015-09-30)entity/entity[</p> {(urn:opendaylight:params:xml:ns:yang:mdsal:core:general-entity?revision= 2015-09-30)name=openflow:1} <p>]], state=LOCAL_OWNERSHIP_LOST_NEW_OWNER <span class="error">[wasOwner=true, isOwner=false, hasOwner=true]</span>, inJeopardy=false]<br/> 2017-09-15 00:21:14,895 | DEBUG | lt-dispatcher-24 | ClusterSingletonServiceGroupImpl | 262 - org.opendaylight.mdsal.singleton-dom-impl - 2.3.0.SNAPSHOT | Ownership change DOMEntityOwnershipChange [entity=DOMEntity [type=org.opendaylight<br/> .mdsal.ServiceEntityType, id=/(urn:opendaylight:params:xml:ns:yang:mdsal:core:general-entity?revision=2015-09-30)entity/entity[</p> {(urn:opendaylight:params:xml:ns:yang:mdsal:core:general-entity?revision=2015-09-30)name=openflow:1} <p>]], state=<br/> LOCAL_OWNERSHIP_LOST_NEW_OWNER <span class="error">[wasOwner=true, isOwner=false, hasOwner=true]</span>, inJeopardy=false] for ClusterSingletonServiceGroup openflow:1<br/> 2017-09-15 00:21:14,895 | DEBUG | lt-dispatcher-24 | ClusterSingletonServiceGroupImpl | 262 - org.opendaylight.mdsal.singleton-dom-impl - 2.3.0.SNAPSHOT | Service group openflow:1 lost ownership in state OWNER<br/> 2017-09-15 00:21:14,895 | DEBUG | lt-dispatcher-24 | ClusterSingletonServiceGroupImpl | 262 - org.opendaylight.mdsal.singleton-dom-impl - 2.3.0.SNAPSHOT | Service group openflow:1 switching from OWNER to STOPPING_SERVICES<br/> 2017-09-15 00:21:14,896 | DEBUG | lt-dispatcher-24 | sterSingletonServiceProviderImpl | 262 - org.opendaylight.mdsal.singleton-dom-impl - 2.3.0.SNAPSHOT | Ownership change for ClusterSingletonService Provider DOMEntityOwnershipChange [en<br/> tity=DOMEntity [type=org.opendaylight.mdsal.AsyncServiceCloseEntityType, id=/(urn:opendaylight:params:xml:ns:yang:mdsal:core:general-entity?revision=2015-09-30)entity/entity[</p> {(urn:opendaylight:params:xml:ns:yang:mdsal:core:general-entity ?revision=2015-09-30)name=openflow:1} <p>]], state=LOCAL_OWNERSHIP_LOST_NEW_OWNER <span class="error">[wasOwner=true, isOwner=false, hasOwner=true]</span>, inJeopardy=false]<br/> 2017-09-15 00:21:14,896 | DEBUG | lt-dispatcher-24 | ClusterSingletonServiceGroupImpl | 262 - org.opendaylight.mdsal.singleton-dom-impl - 2.3.0.SNAPSHOT | Ownership change DOMEntityOwnershipChange [entity=DOMEntity [type=org.opendaylight<br/> .mdsal.AsyncServiceCloseEntityType, id=/(urn:opendaylight:params:xml:ns:yang:mdsal:core:general-entity?revision=2015-09-30)entity/entity[</p> {(urn:opendaylight:params:xml:ns:yang:mdsal:core:general-entity?revision=2015-09-30)name=openflow:1} <p>]], state=LOCAL_OWNERSHIP_LOST_NEW_OWNER <span class="error">[wasOwner=true, isOwner=false, hasOwner=true]</span>, inJeopardy=false] for ClusterSingletonServiceGroup openflow:1<br/> 2017-09-15 00:21:14,896 | DEBUG | lt-dispatcher-24 | ClusterSingletonServiceGroupImpl | 262 - org.opendaylight.mdsal.singleton-dom-impl - 2.3.0.SNAPSHOT | Group openflow:1 in state STOPPING_SERVICES ignoring cleanup OwnershipChange DOMEn<br/> tityOwnershipChange [entity=DOMEntity [type=org.opendaylight.mdsal.AsyncServiceCloseEntityType, id=/(urn:opendaylight:params:xml:ns:yang:mdsal:core:general-entity?revision=2015-09-30)entity/entity[</p> {(urn:opendaylight:params:xml:ns:yang:md sal:core:general-entity?revision=2015-09-30)name=openflow:1} <p>]], state=LOCAL_OWNERSHIP_LOST_NEW_OWNER <span class="error">[wasOwner=true, isOwner=false, hasOwner=true]</span>, inJeopardy=false]<br/> 2017-09-15 00:21:14,897 | DEBUG | ofppool-0 | ClusterSingletonServiceGroupImpl | 262 - org.opendaylight.mdsal.singleton-dom-impl - 2.3.0.SNAPSHOT | Service group openflow:1 finished stopping services<br/> 2017-09-15 00:21:14,897 | DEBUG | ofppool-0 | ClusterSingletonServiceGroupImpl | 262 - org.opendaylight.mdsal.singleton-dom-impl - 2.3.0.SNAPSHOT | Service group openflow:1 switching from STOPPING_SERVICES to RELEASING_OWNERSHIP<br/> 2017-09-15 00:23:09,057 | DEBUG | lt-dispatcher-21 | sterSingletonServiceProviderImpl | 262 - org.opendaylight.mdsal.singleton-dom-impl - 2.3.0.SNAPSHOT | Ownership change for ClusterSingletonService Provider DOMEntityOwnershipChange [entity=DOMEntity [type=org.opendaylight.mdsal.ServiceEntityType, id=/(urn:opendaylight:params:xml:ns:yang:mdsal:core:general-entity?revision=2015-09-30)entity/entity[</p> {(urn:opendaylight:params:xml:ns:yang:mdsal:core:general-entity?revision=2015-09-30)name=openflow:1} <p>]], state=REMOTE_OWNERSHIP_LOST_NO_OWNER <span class="error">[wasOwner=false, isOwner=false, hasOwner=false]</span>, inJeopardy=false]<br/> 2017-09-15 00:23:09,057 | DEBUG | lt-dispatcher-21 | ClusterSingletonServiceGroupImpl | 262 - org.opendaylight.mdsal.singleton-dom-impl - 2.3.0.SNAPSHOT | Ownership change DOMEntityOwnershipChange [entity=DOMEntity [type=org.opendaylight.mdsal.ServiceEntityType, id=/(urn:opendaylight:params:xml:ns:yang:mdsal:core:general-entity?revision=2015-09-30)entity/entity[</p> {(urn:opendaylight:params:xml:ns:yang:mdsal:core:general-entity?revision=2015-09-30)name=openflow:1} <p>]], state=REMOTE_OWNERSHIP_LOST_NO_OWNER <span class="error">[wasOwner=false, isOwner=false, hasOwner=false]</span>, inJeopardy=false] for ClusterSingletonServiceGroup openflow:1<br/> 2017-09-15 00:23:09,057 | DEBUG | lt-dispatcher-21 | ClusterSingletonServiceGroupImpl | 262 - org.opendaylight.mdsal.singleton-dom-impl - 2.3.0.SNAPSHOT | Group openflow:1 in state RELEASING_OWNERSHIP not processed entity OwnershipChange DOMEntityOwnershipChange [entity=DOMEntity [type=org.opendaylight.mdsal.ServiceEntityType, id=/(urn:opendaylight:params:xml:ns:yang:mdsal:core:general-entity?revision=2015-09-30)entity/entity[</p> {(urn:opendaylight:params:xml:ns:yang:mdsal:core:general-entity?revision=2015-09-30)name=openflow:1} <p>]], state=REMOTE_OWNERSHIP_LOST_NO_OWNER <span class="error">[wasOwner=false, isOwner=false, hasOwner=false]</span>, inJeopardy=false]</p> <p>I will review that in detail later this week.</p> <p>> downgrading priority</p> <p>Looking at <span class="error">[11]</span> section 5.q it seems we want to keep this as a blocker (so we can merge a fix) until this is the last blocking bug, then we downgrade this and build a (hopefully) final RC build.</p> <p><span class="error">[11]</span> <a href="https://meetings.opendaylight.org/opendaylight-meeting/2017/tsc/opendaylight-meeting-tsc.2017-09-14-17.02.html" class="external-link" target="_blank" rel="nofollow noopener">https://meetings.opendaylight.org/opendaylight-meeting/2017/tsc/opendaylight-meeting-tsc.2017-09-14-17.02.html</a></p> <p>So the issue occurs after EOS leadership election, when SingletonService forcibly loses ownership:</p> <p>ber-2-shard-entity-ownership-operational, leaderPayloadVersion=5]<br/> 2017-09-15 00:21:14,892 | INFO | rd-dispatcher-27 | ShardManager | 211 - org.opendaylight.controller.sal-distributed-datastore - 1.6.0.SNAPSHOT | shard-manager-operational Received follower initial sync status for member-1-shard-entity-ownership-operational status sync done false<br/> 2017-09-15 00:21:14,895 | DEBUG | lt-dispatcher-24 | sterSingletonServiceProviderImpl | 262 - org.opendaylight.mdsal.singleton-dom-impl - 2.3.0.SNAPSHOT | Ownership change for ClusterSingletonService Provider DOMEntityOwnershipChange [entity=DOMEntity [type=org.opendaylight.mdsal.ServiceEntityType, id=/(urn:opendayl<br/> ight:params:xml:ns:yang:mdsal:core:general-entity?revision=2015-09-30)entity/entity[</p> {(urn:opendaylight:params:xml:ns:yang:mdsal:core:general-entity?revision=2015-09-30)name=openflow:1} <p>]], state=LOCAL_OWNERSHIP_LOST_NEW_OWNER <span class="error">[wasOwner=true, isOwner=false, hasOwner=true]</span>, inJeopardy=false]<br/> 2017-09-15 00:21:14,895 | DEBUG | lt-dispatcher-24 | ClusterSingletonServiceGroupImpl | 262 - org.opendaylight.mdsal.singleton-dom-impl - 2.3.0.SNAPSHOT | Ownership change DOMEntityOwnershipChange [entity=DOMEntity [type=org.opendaylight.mdsal.ServiceEntityType, id=/(urn:opendaylight:params:xml:ns:yang:mdsal:core:ge<br/> neral-entity?revision=2015-09-30)entity/entity[</p> {(urn:opendaylight:params:xml:ns:yang:mdsal:core:general-entity?revision=2015-09-30)name=openflow:1} <p>]], state=LOCAL_OWNERSHIP_LOST_NEW_OWNER <span class="error">[wasOwner=true, isOwner=false, hasOwner=true]</span>, inJeopardy=false] for ClusterSingletonServiceGroup openflow:1<br/> 2017-09-15 00:21:14,895 | DEBUG | lt-dispatcher-24 | ClusterSingletonServiceGroupImpl | 262 - org.opendaylight.mdsal.singleton-dom-impl - 2.3.0.SNAPSHOT | Service group openflow:1 lost ownership in state OWNER<br/> 2017-09-15 00:21:14,895 | DEBUG | lt-dispatcher-24 | ClusterSingletonServiceGroupImpl | 262 - org.opendaylight.mdsal.singleton-dom-impl - 2.3.0.SNAPSHOT | Service group openflow:1 switching from OWNER to STOPPING_SERVICES</p> <p>This is a slightly different codepath, which is then not recovered and gets stuck in RELEASING_OWNERSHIP – the state machine should be skipping that step, as we already lost ownership, hence we should transition to STANDBY instead.</p> <p>> This week, I got</p> <p>Run #11 <span class="error">[12]</span> finally looks like the original symptom. Specifically, member-1 was owner, git hit by owner isolation. After that, member-1 was no longer an owner, so it got hit again by successor isolation. As there was no progress during the seconf isolation, and member-1 was initiating elections with higher term, member-1 become device owner after second rejoin.</p> <p>> gets stuck in RELEASING_OWNERSHIP</p> <p>While member-1 got stuck after the first rejoin, it unstuck after becoming owner again. It was member-2, stuck after losing ownership at the second rejoin, who got stuck indefinitely.</p> <p>Looking at the karaf.log <span class="error">[13]</span> of member-2, that (getting stuck after the successor rejoin) is the cause of this bug indeed.</p> <p>2017-09-20 08:36:02,167 | DEBUG | ult-dispatcher-4 | ClusterSingletonServiceGroupImpl | 262 - org.opendaylight.mdsal.singleton-dom-impl - 2.3.0.SNAPSHOT | Ownership change DOMEntityOwnershipChange [entity=DOMEntity [type=org.opendaylight.mdsal.ServiceEntityType, id=/(urn:opendaylight:params:xml:ns:yang:mdsal:core:general-entity?revision=2015-09-30)entity/entity[</p> {(urn:opendaylight:params:xml:ns:yang:mdsal:core:general-entity?revision=2015-09-30)name=openflow:1} <p>]], state=LOCAL_OWNERSHIP_LOST_NEW_OWNER <span class="error">[wasOwner=true, isOwner=false, hasOwner=true]</span>, inJeopardy=false] for ClusterSingletonServiceGroup openflow:1<br/> 2017-09-20 08:36:02,167 | DEBUG | ult-dispatcher-4 | ClusterSingletonServiceGroupImpl | 262 - org.opendaylight.mdsal.singleton-dom-impl - 2.3.0.SNAPSHOT | Service group openflow:1 lost ownership in state OWNER<br/> 2017-09-20 08:36:02,167 | DEBUG | ult-dispatcher-4 | ClusterSingletonServiceGroupImpl | 262 - org.opendaylight.mdsal.singleton-dom-impl - 2.3.0.SNAPSHOT | Service group openflow:1 switching from OWNER to STOPPING_SERVICES<br/> ...<br/> 2017-09-20 08:36:02,175 | DEBUG | ofppool-0 | ClusterSingletonServiceGroupImpl | 262 - org.opendaylight.mdsal.singleton-dom-impl - 2.3.0.SNAPSHOT | Service group openflow:1 switching from STOPPING_SERVICES to RELEASING_OWNERSHIP<br/> ...<br/> 2017-09-20 08:36:15,920 | INFO | entLoopGroup-7-2 | SystemNotificationsListenerImpl | 281 - org.opendaylight.openflowplugin.impl - 0.5.0.SNAPSHOT | ConnectionEvent: Connection closed by device, Device:/10.29.12.136:35320, NodeId:openflow:1<br/> ...<br/> 2017-09-20 08:36:15,921 | DEBUG | entLoopGroup-7-2 | sterSingletonServiceProviderImpl | 262 - org.opendaylight.mdsal.singleton-dom-impl - 2.3.0.SNAPSHOT | Replaced group openflow:1 with PlaceholderGroup</p> {id=openflow:1} <p>...<br/> 2017-09-20 08:36:15,943 | DEBUG | lt-dispatcher-34 | ClusterSingletonServiceGroupImpl | 262 - org.opendaylight.mdsal.singleton-dom-impl - 2.3.0.SNAPSHOT | Service group openflow:1 is closed, not taking ownership<br/> ...<br/> 2017-09-20 08:36:15,943 | DEBUG | lt-dispatcher-34 | ClusterSingletonServiceGroupImpl | 262 - org.opendaylight.mdsal.singleton-dom-impl - 2.3.0.SNAPSHOT | Service group openflow:1 lost ownership in state RELEASING_OWNERSHIP<br/> 2017-09-20 08:36:15,943 | INFO | lt-dispatcher-34 | ClusterSingletonServiceGroupImpl | 262 - org.opendaylight.mdsal.singleton-dom-impl - 2.3.0.SNAPSHOT | Service group openflow:1 ignoring lost ownership in state RELEASING_OWNERSHIP,<br/> ...<br/> 2017-09-20 08:36:15,950 | INFO | ult-dispatcher-5 | ContextChainHolderImpl | 281 - org.opendaylight.openflowplugin.impl - 0.5.0.SNAPSHOT | Removing device from operational DS Uri <span class="error">[_value=openflow:1]</span> was successful<br/> 2017-09-20 08:36:21,485 | INFO | entLoopGroup-9-1 | ConnectionAdapterImpl | 287 - org.opendaylight.openflowplugin.openflowjava.openflow-protocol-impl - 0.5.0.SNAPSHOT | Hello received<br/> ...<br/> 2017-09-20 08:36:21,489 | INFO | entLoopGroup-9-1 | RoleContextImpl | 281 - org.opendaylight.openflowplugin.impl - 0.5.0.SNAPSHOT | Started timer for setting SLAVE role on device openflow:1 if no role will be set in 20s.<br/> 2017-09-20 08:36:21,489 | DEBUG | entLoopGroup-9-1 | sterSingletonServiceProviderImpl | 262 - org.opendaylight.mdsal.singleton-dom-impl - 2.3.0.SNAPSHOT | Call registrationService org.opendaylight.openflowplugin.impl.lifecycle.ContextChainImpl@14379d5c method for ClusterSingletonService Provider org.opendaylight.mdsal.singleton.dom.impl.DOMClusterSingletonServiceProviderImpl@67124196<br/> 2017-09-20 08:36:21,489 | DEBUG | entLoopGroup-9-1 | PlaceholderGroup | 262 - org.opendaylight.mdsal.singleton-dom-impl - 2.3.0.SNAPSHOT | PlaceholderGroup</p> {id=openflow:1} <p>: added service org.opendaylight.openflowplugin.impl.lifecycle.ContextChainImpl@14379d5c<br/> ...<br/> 2017-09-20 08:36:21,496 | DEBUG | lt-dispatcher-34 | ClusterSingletonServiceGroupImpl | 262 - org.opendaylight.mdsal.singleton-dom-impl - 2.3.0.SNAPSHOT | Group openflow:1 in state RELEASING_OWNERSHIP not processed entity OwnershipChange DOMEntityOwnershipChange [entity=DOMEntity [type=org.opendaylight.mdsal.ServiceEntityType, id=/(urn:opendaylight:params:xml:ns:yang:mdsal:core:general-entity?revision=2015-09-30)entity/entity[</p> {(urn:opendaylight:params:xml:ns:yang:mdsal:core:general-entity?revision=2015-09-30)name=openflow:1} <p>]], state=REMOTE_OWNERSHIP_CHANGED <span class="error">[wasOwner=false, isOwner=false, hasOwner=true]</span>, inJeopardy=false]<br/> ...<br/> 2017-09-20 08:36:41,499 | INFO | entLoopGroup-9-1 | ContextChainHolderImpl | 281 - org.opendaylight.openflowplugin.impl - 0.5.0.SNAPSHOT | Role SLAVE was granted to device openflow:1</p> <p>> we should transition to STANDBY instead</p> <p>Yes, that would fix this. More specifically, we need to split STOPPING_SERVICES into (something like) STOPPING_SERVICES_BEFORE_UNREGISTERING and STOPPING_SERVICES_AFTER_LOSING_OWNERSHIP.</p> <p><span class="error">[12]</span> <a href="https://jenkins.opendaylight.org/sandbox/job/openflowplugin-csit-3node-clustering-only-nitrogen/11/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/sandbox/job/openflowplugin-csit-3node-clustering-only-nitrogen/11/</a><br/> <span class="error">[13]</span> <a href="https://logs.opendaylight.org/sandbox/jenkins091/openflowplugin-csit-3node-clustering-only-nitrogen/11/odl2_karaf.log.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/sandbox/jenkins091/openflowplugin-csit-3node-clustering-only-nitrogen/11/odl2_karaf.log.gz</a></p> <p>Given the root cause is identified, what is the ETA for a patch? I guess depending on this ETA we can downgrade this bug or hold for a while.</p> <p>I am still working through the state transitions, ETA today late evening or tomorrow – depending on how it goes.</p> <p>So I think the agreement is to fix this for Carbon SR3 and therefore Nitrogen SR1 too.</p> <p>Yes, slight correction: Carbon SR2, not SR3 (that would be way out <img class="emoticon" src="https://jira.opendaylight.org/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/>)</p> <p>preliminary patch: <a href="https://git.opendaylight.org/gerrit/63426" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/63426</a></p> <p>needs some more work to resolve the implementation/UT mismatch (7 failing tests).</p> <p>Right SR2, I thought next was SR3...</p> <p>I tried to reproduce this withing controller clustering job, by extending <span class="error">[15]</span> existing "singleton partition and heal" suite, but so far no luck replicating this Bug there (the one failure on this week's Sandbox <span class="error">[16]</span> is <a href="https://jira.opendaylight.org/browse/CONTROLLER-1670" title="Rejoining member can make other member forget remotely registered RPC" class="issue-link" data-issue-key="CONTROLLER-1670"><del>CONTROLLER-1670</del></a>).</p> <p>> preliminary patch: <a href="https://git.opendaylight.org/gerrit/63426" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/63426</a></p> <p>That Oxygen change already makes the Openflow scenario more stable.<br/> From this week's Sandbox job <span class="error">[17]</span>, so far only two failures happened (#2 and #18, both times member-3 was not completely removed from owners upon device disconnect).<br/> Odd numbered runs are without this change, mostly failing (although I have not checked the exact cause each time).</p> <p><span class="error">[15]</span> <a href="https://git.opendaylight.org/gerrit/63492" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/63492</a><br/> <span class="error">[16]</span> <a href="https://jenkins.opendaylight.org/sandbox/job/controller-csit-3node-clustering-only-nitrogen/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/sandbox/job/controller-csit-3node-clustering-only-nitrogen/</a><br/> <span class="error">[17]</span> <a href="https://jenkins.opendaylight.org/sandbox/job/openflowplugin-csit-3node-clustering-only-oxygen/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/sandbox/job/openflowplugin-csit-3node-clustering-only-oxygen/</a></p> <p>What's the status on this one?</p> <p>I believe it's the only bug remaining holding up Carbon SR2.</p> <p>I believe the cherry-pick to Carbon <span class="error">[18]</span> (with its parent) is good enough for SR2.<br/> But Robert may have other plans.</p> <p><span class="error">[18]</span> <a href="https://git.opendaylight.org/gerrit/#/c/63866/2" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/63866/2</a></p> <p>Please verify this issue is fixed in the latest Carbon SR2 build. Thanks!</p> <p>Yes this is fixed now:</p> <p><a href="https://jenkins.opendaylight.org/releng/view/CSIT-3node/job/openflowplugin-csit-3node-clustering-only-carbon/783/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/CSIT-3node/job/openflowplugin-csit-3node-clustering-only-carbon/783/</a></p> <p>Hi Luis,</p> <p>In the latest successful nitrogen autorelease build 242, this same CSIT job is successful:</p> <p><a href="https://jenkins.opendaylight.org/releng/view/autorelease/job/openflowplugin-csit-3node-clustering-only-nitrogen/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/autorelease/job/openflowplugin-csit-3node-clustering-only-nitrogen/</a></p> <p>Does this mean we are good with this bug for nitrogen SR1? Thanks!</p> <p>AFAIK fix was merged in all branches so yes we are good.</p> Development External issue ID 9145 External issue URL Issue Type ODL SR Target Milestone Rank 0|i02xhr: