https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/CONTROLLER-1895 controller <p>There is a test where we set 1 switch-switch link down in cluster deployment, and the switches belong to separate instances.</p> <p>The test works with ask-protocol:</p> <p><a href="https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-3node-gate-clustering-only-sodium/71/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-3node-gate-clustering-only-sodium/71/</a></p> <p>But it fails with tell-based (links are not removed from topology):</p> <p><a href="https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-3node-gate-clustering-only-sodium/72/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-3node-gate-clustering-only-sodium/72/</a></p> <p>After enabling some DEBUGs, I can see 7 TP notifications with ask-based protocol:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> 2019-05-20T20:14:25,096 | DEBUG | ofp-topo-expo-flow:1 | OperationProcessor | 305 - org.opendaylight.openflowplugin.applications.topology-manager - 0.9.0.SNAPSHOT | New org.opendaylight.openflowplugin.applications.topology.manager.DataTreeChangeListenerImpl$$Lambda$1725/789675131@26eac440 operation available, starting transaction 2019-05-20T20:14:25,098 | DEBUG | ofp-topo-expo-flow:1 | OperationProcessor | 305 - org.opendaylight.openflowplugin.applications.topology-manager - 0.9.0.SNAPSHOT | Next operation org.opendaylight.openflowplugin.applications.topology.manager.TerminationPointChangeListenerImpl$$Lambda$1726/1648087628@2057bbed 2019-05-20T20:14:25,098 | DEBUG | ofp-topo-expo-flow:1 | OperationProcessor | 305 - org.opendaylight.openflowplugin.applications.topology-manager - 0.9.0.SNAPSHOT | Next operation org.opendaylight.openflowplugin.applications.topology.manager.DataTreeChangeListenerImpl$$Lambda$1725/789675131@71bc860a 2019-05-20T20:14:25,099 | DEBUG | ofp-topo-expo-flow:1 | OperationProcessor | 305 - org.opendaylight.openflowplugin.applications.topology-manager - 0.9.0.SNAPSHOT | Next operation org.opendaylight.openflowplugin.applications.topology.manager.TerminationPointChangeListenerImpl$$Lambda$1726/1648087628@6c667215 2019-05-20T20:14:25,106 | DEBUG | ofp-topo-expo-flow:1 | OperationProcessor | 305 - org.opendaylight.openflowplugin.applications.topology-manager - 0.9.0.SNAPSHOT | New org.opendaylight.openflowplugin.applications.topology.manager.DataTreeChangeListenerImpl$$Lambda$1725/789675131@1cde631d operation available, starting transaction 2019-05-20T20:14:25,107 | DEBUG | ofp-topo-expo-flow:1 | OperationProcessor | 305 - org.opendaylight.openflowplugin.applications.topology-manager - 0.9.0.SNAPSHOT | Next operation org.opendaylight.openflowplugin.applications.topology.manager.TerminationPointChangeListenerImpl$$Lambda$1726/1648087628@14b2bb8 2019-05-20T20:14:25,110 | DEBUG | ofp-topo-expo-flow:1 | OperationProcessor | 305 - org.opendaylight.openflowplugin.applications.topology-manager - 0.9.0.SNAPSHOT | New org.opendaylight.openflowplugin.applications.topology.manager.DataTreeChangeListenerImpl$$Lambda$1725/789675131@1452614d operation available, starting transaction 2019-05-20T20:14:25,112 | DEBUG | ofp-topo-expo-flow:1 | OperationProcessor | 305 - org.opendaylight.openflowplugin.applications.topology-manager - 0.9.0.SNAPSHOT | Next operation org.opendaylight.openflowplugin.applications.topology.manager.TerminationPointChangeListenerImpl$$Lambda$1726/1648087628@7c1cf238 2019-05-20T20:14:25,515 | DEBUG | ofp-topo-expo-flow:1 | OperationProcessor | 305 - org.opendaylight.openflowplugin.applications.topology-manager - 0.9.0.SNAPSHOT | New org.opendaylight.openflowplugin.applications.topology.manager.DataTreeChangeListenerImpl$$Lambda$1725/789675131@7bc3acff operation available, starting transaction 2019-05-20T20:14:25,516 | DEBUG | ofp-topo-expo-flow:1 | OperationProcessor | 305 - org.opendaylight.openflowplugin.applications.topology-manager - 0.9.0.SNAPSHOT | Next operation org.opendaylight.openflowplugin.applications.topology.manager.TerminationPointChangeListenerImpl$$Lambda$1726/1648087628@6ad1bf80 2019-05-20T20:14:25,522 | DEBUG | ofp-topo-expo-flow:1 | OperationProcessor | 305 - org.opendaylight.openflowplugin.applications.topology-manager - 0.9.0.SNAPSHOT | New org.opendaylight.openflowplugin.applications.topology.manager.DataTreeChangeListenerImpl$$Lambda$1725/789675131@5ee83cb8 operation available, starting transaction 2019-05-20T20:14:25,523 | DEBUG | ofp-topo-expo-flow:1 | OperationProcessor | 305 - org.opendaylight.openflowplugin.applications.topology-manager - 0.9.0.SNAPSHOT | Next operation org.opendaylight.openflowplugin.applications.topology.manager.TerminationPointChangeListenerImpl$$Lambda$1726/1648087628@45fe986f 2019-05-20T20:14:25,524 | DEBUG | ofp-topo-expo-flow:1 | OperationProcessor | 305 - org.opendaylight.openflowplugin.applications.topology-manager - 0.9.0.SNAPSHOT | New org.opendaylight.openflowplugin.applications.topology.manager.DataTreeChangeListenerImpl$$Lambda$1725/789675131@23bc607b operation available, starting transaction 2019-05-20T20:14:25,525 | DEBUG | ofp-topo-expo-flow:1 | OperationProcessor | 305 - org.opendaylight.openflowplugin.applications.topology-manager - 0.9.0.SNAPSHOT | Next operation org.opendaylight.openflowplugin.applications.topology.manager.TerminationPointChangeListenerImpl$$Lambda$1726/1648087628@44f6e633 </pre> </div></div> <p>And only 4 with tell-based protocol:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> 2019-05-20T20:19:51,259 | DEBUG | ofp-topo-expo-flow:1 | OperationProcessor | 305 - org.opendaylight.openflowplugin.applications.topology-manager - 0.9.0.SNAPSHOT | New org.opendaylight.openflowplugin.applications.topology.manager.DataTreeChangeListenerImpl$$Lambda$1808/2124882347@64c577c1 operation available, starting transaction 2019-05-20T20:19:51,260 | DEBUG | ofp-topo-expo-flow:1 | OperationProcessor | 305 - org.opendaylight.openflowplugin.applications.topology-manager - 0.9.0.SNAPSHOT | Next operation org.opendaylight.openflowplugin.applications.topology.manager.TerminationPointChangeListenerImpl$$Lambda$1814/1542219333@7095f28a 2019-05-20T20:19:51,261 | DEBUG | ofp-topo-expo-flow:1 | OperationProcessor | 305 - org.opendaylight.openflowplugin.applications.topology-manager - 0.9.0.SNAPSHOT | New org.opendaylight.openflowplugin.applications.topology.manager.DataTreeChangeListenerImpl$$Lambda$1808/2124882347@3fa1e826 operation available, starting transaction 2019-05-20T20:19:51,262 | DEBUG | ofp-topo-expo-flow:1 | OperationProcessor | 305 - org.opendaylight.openflowplugin.applications.topology-manager - 0.9.0.SNAPSHOT | Next operation org.opendaylight.openflowplugin.applications.topology.manager.TerminationPointChangeListenerImpl$$Lambda$1814/1542219333@4477848c 2019-05-20T20:19:51,264 | DEBUG | ofp-topo-expo-flow:1 | OperationProcessor | 305 - org.opendaylight.openflowplugin.applications.topology-manager - 0.9.0.SNAPSHOT | New org.opendaylight.openflowplugin.applications.topology.manager.DataTreeChangeListenerImpl$$Lambda$1808/2124882347@7427b4fc operation available, starting transaction 2019-05-20T20:19:51,265 | DEBUG | ofp-topo-expo-flow:1 | OperationProcessor | 305 - org.opendaylight.openflowplugin.applications.topology-manager - 0.9.0.SNAPSHOT | Next operation org.opendaylight.openflowplugin.applications.topology.manager.TerminationPointChangeListenerImpl$$Lambda$1814/1542219333@55aa94cf 2019-05-20T20:19:51,267 | DEBUG | ofp-topo-expo-flow:1 | OperationProcessor | 305 - org.opendaylight.openflowplugin.applications.topology-manager - 0.9.0.SNAPSHOT | New org.opendaylight.openflowplugin.applications.topology.manager.DataTreeChangeListenerImpl$$Lambda$1808/2124882347@1ce34ad2 operation available, starting transaction 2019-05-20T20:19:51,268 | DEBUG | ofp-topo-expo-flow:1 | OperationProcessor | 305 - org.opendaylight.openflowplugin.applications.topology-manager - 0.9.0.SNAPSHOT | Next operation org.opendaylight.openflowplugin.applications.topology.manager.TerminationPointChangeListenerImpl$$Lambda$1814/1542219333@26e81ce5 2019-05-20T20:19:51,268 | DEBUG | ofp-topo-expo-flow:1 | OperationProcessor | 305 - org.opendaylight.openflowplugin.applications.topology-manager - 0.9.0.SNAPSHOT | Next operation <span class="code-keyword">null</span> </pre> </div></div> CONTROLLER-1895

Links not removed from topology with tell-based

Bug High Open Unresolved Luis Gomez Luis Gomez Mon, 20 May 2019 21:17:16 +0000 Tue, 1 Oct 2019 20:23:03 +0000 clustering 0 3 <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=Avishnoi" class="user-hover" rel="Avishnoi">Avishnoi</a>, can you shed some light here? what kind of notifications are these and where are in the OFP code?</p> <p>Following class is registering listener to get  notification on FlowCapableNodeConnector</p> <p> </p> <p><a href="https://github.com/opendaylight/openflowplugin/blob/master/applications/topology-manager/src/main/java/org/opendaylight/openflowplugin/applications/topology/manager/TerminationPointChangeListenerImpl.java#L45" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/opendaylight/openflowplugin/blob/master/applications/topology-manager/src/main/java/org/opendaylight/openflowplugin/applications/topology/manager/TerminationPointChangeListenerImpl.java#L45</a></p> <p> </p> <p>It processes the DTCL and adds the task to the queue in OperationProcessor.</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=ecelgp" class="user-hover" rel="ecelgp">ecelgp</a> We need to dump the DTCL output in this class for both the scenario. If there is a diff in the way DTCN is published by datastore, that will give some concrete information for robert to debug it further.</p> <p>I pushed a patch to openflowplugin master branch </p> <p> </p> <p><a href="https://git.opendaylight.org/gerrit/#/c/82374/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/82374/</a></p> <p> </p> <p>Can you run the job with this patch and with both the tell and ask protocol? In the log, grep for <a href="https://jira.opendaylight.org/browse/CONTROLLER-1895" title="Links not removed from topology with tell-based" class="issue-link" data-issue-key="CONTROLLER-1895">CONTROLLER-1895</a> to grep the data tree change notification.</p> <p> </p> <p>After running the test with the OFP patch I can see the notification # discrepancy is OK: sometimes controller gets single DS notification containing "link-down=true" &amp; "configuration=PORT-DOWN" while other times it gets 2 notifications with one change on each.</p> <p>So if the notification is not the problem, what is next DEBUG to figure out why port down does not remove link from topology?</p> <p>Either case should be fine: <a href="https://github.com/opendaylight/openflowplugin/blob/master/applications/topology-manager/src/main/java/org/opendaylight/openflowplugin/applications/topology/manager/TerminationPointChangeListenerImpl.java#L131" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/opendaylight/openflowplugin/blob/master/applications/topology-manager/src/main/java/org/opendaylight/openflowplugin/applications/topology/manager/TerminationPointChangeListenerImpl.java#L131</a></p> <p>I wonder whether we are not in SUBTREE_MODIFIED case which ends up landing here:</p> <p><a href="https://github.com/opendaylight/openflowplugin/blob/master/applications/topology-manager/src/main/java/org/opendaylight/openflowplugin/applications/topology/manager/TerminationPointChangeListenerImpl.java#L111" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/opendaylight/openflowplugin/blob/master/applications/topology-manager/src/main/java/org/opendaylight/openflowplugin/applications/topology/manager/TerminationPointChangeListenerImpl.java#L111</a></p> <p>I  have updated the patch to warn when that happens.</p> <p>I re-run the test with new patchset:</p> <p><a href="https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-3node-gate-clustering-only-sodium/82/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-3node-gate-clustering-only-sodium/82/</a></p> <p>But I do not see the "Unhandled termination point change" in the logs.</p> <p>Alright, we need to move this forward. <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=ecelgp" class="user-hover" rel="ecelgp">ecelgp</a> I remember you mentioning this happening even with ask-based on stable/sodium. I seem to remember timing being a role. Can you confirm?</p> <p>If that is the case, we need a tracker in OFP, because it is likely they have an issue we are unmasking here.</p> Development Rank 0|i03nvz: