https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/OPNFLWPLUG-1039 OpenFlowPlugin <p><a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-3node-0cmb-1ctl-2cmp-openstack-queens-upstream-stateful-oxygen/95/odl_3/odl3_karaf.log.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-3node-0cmb-1ctl-2cmp-openstack-queens-upstream-stateful-oxygen/95/odl_3/odl3_karaf.log.gz</a></p> <p>The exception and errors below are repeated numerous times as ovs nodes are initially connected to ODL. This looks to lead to initial default flows not being programmed on the switches.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>2018-10-07T21:49:21,087 | INFO | pool-83-thread-1 | SalRoleServiceImpl | 382 - org.opendaylight.openflowplugin.impl - 0.6.4 | SetRole called with input:SetRoleInput [_controllerRole=BECOMESLAVE, _node=NodeRef [_value=KeyedInstanceIdentifier{targetType=interface org.opendaylight.yang.gen.v1.urn.opendaylight.inventory.rev130819.nodes.Node, path=[org.opendaylight.yang.gen.v1.urn.opendaylight.inventory.rev130819.Nodes, org.opendaylight.yang.gen.v1.urn.opendaylight.inventory.rev130819.nodes.Node[key=NodeKey [_id=Uri [_value=openflow:97702827111235]]]]}], augmentation=[]] 2018-10-07T21:49:21,088 | INFO | pool-83-thread-1 | SalRoleServiceImpl | 382 - org.opendaylight.openflowplugin.impl - 0.6.4 | Requesting state change to BECOMESLAVE 2018-10-07T21:49:21,088 | INFO | pool-83-thread-1 | SalRoleServiceImpl | 382 - org.opendaylight.openflowplugin.impl - 0.6.4 | RoleChangeTask called on device:openflow:97702827111235 OFPRole:BECOMESLAVE 2018-10-07T21:49:21,088 | INFO | pool-83-thread-1 | RoleService | 382 - org.opendaylight.openflowplugin.impl - 0.6.4 | getGenerationIdFromDevice called for device: openflow:97702827111235 2018-10-07T21:49:21,093 | WARN | pool-83-thread-1 | OutboundQueueEntry | 391 - org.opendaylight.openflowplugin.openflowjava.openflow-protocol-impl - 0.6.4 | Can't commit a completed message. 2018-10-07T21:49:21,096 | ERROR | pool-83-thread-1 | RoleService | 382 - org.opendaylight.openflowplugin.impl - 0.6.4 | getGenerationIdFromDevice RPC error null 2018-10-07T21:49:21,099 | WARN | pool-83-thread-1 | ContextChainHolderImpl | 382 - org.opendaylight.openflowplugin.impl - 0.6.4 | Not able to set SLAVE role on device openflow:97702827111235, reason: Was not able to propagate SLAVE role on device. Error: org.opendaylight.openflowplugin.impl.role.RoleChangeException: Can't commit a completed message. 2018-10-07T21:49:21,100 | INFO | pool-83-thread-1 | ReconciliationManagerImpl | 376 - org.opendaylight.openflowplugin.applications.reconciliation-framework - 0.6.4 | Stopping reconciliation for node Uri [_value=openflow:97702827111235] 2018-10-07T21:49:21,105 | INFO | pool-83-thread-1 | ContextChainImpl | 382 - org.opendaylight.openflowplugin.impl - 0.6.4 | Closed clustering services registration for node openflow:97702827111235 2018-10-07T21:49:21,106 | INFO | pool-83-thread-1 | GuardedContextImpl | 382 - org.opendaylight.openflowplugin.impl - 0.6.4 | Terminating DeviceContextImpl[NEW] service for node openflow:97702827111235 2018-10-07T21:49:21,106 | INFO | pool-83-thread-1 | GuardedContextImpl | 382 - org.opendaylight.openflowplugin.impl - 0.6.4 | Terminating RpcContextImpl[NEW] service for node openflow:97702827111235 2018-10-07T21:49:21,107 | INFO | pool-83-thread-1 | GuardedContextImpl | 382 - org.opendaylight.openflowplugin.impl - 0.6.4 | Terminating StatisticsContextImpl[NEW] service for node openflow:97702827111235 2018-10-07T21:49:21,107 | INFO | pool-83-thread-1 | StatisticsContextImpl | 382 - org.opendaylight.openflowplugin.impl - 0.6.4 | Stopping running statistics gathering for node openflow:97702827111235 2018-10-07T21:49:21,113 | INFO | pool-83-thread-1 | GuardedContextImpl | 382 - org.opendaylight.openflowplugin.impl - 0.6.4 | Terminating RoleContextImpl[NEW] service for node openflow:97702827111235 2018-10-07T21:49:21,190 | INFO | epollEventLoopGroup-9-15 | AbstractConnectionAdapter | 391 - org.opendaylight.openflowplugin.openflowjava.openflow-protocol-impl - 0.6.4 | The channel outbound queue size:1024 2018-10-07T21:49:21,191 | INFO | epollEventLoopGroup-9-15 | ConnectionAdapterImpl | 391 - org.opendaylight.openflowplugin.openflowjava.openflow-protocol-impl - 0.6.4 | Hello received 2018-10-07T21:49:21,196 | INFO | epollEventLoopGroup-9-15 | ContextChainHolderImpl | 382 - org.opendaylight.openflowplugin.impl - 0.6.4 | Device openflow:97702827111235 connected. 2018-10-07T21:49:21,197 | INFO | epollEventLoopGroup-9-15 | ContextChainHolderImpl | 382 - org.opendaylight.openflowplugin.impl - 0.6.4 | No context chain found for device: openflow:97702827111235, creating new. 2018-10-07T21:49:21,197 | INFO | epollEventLoopGroup-9-15 | DeviceManagerImpl | 382 - org.opendaylight.openflowplugin.impl - 0.6.4 | ConnectionEvent: Device connected to controller, Device:/10.30.170.126:57622, NodeId:Uri [_value=openflow:97702827111235] 2018-10-07T21:49:21,198 | INFO | epollEventLoopGroup-9-15 | RoleContextImpl | 382 - org.opendaylight.openflowplugin.impl - 0.6.4 | Started timer for setting SLAVE role on device openflow:97702827111235 if no role will be set in 20s. </pre> </div></div> OPNFLWPLUG-1039

Not able to set SLAVE role on device openflow:97702827111235, reason: Was not able to propagate SLAVE role on device. Error: org.opendaylight.openflowplugin.impl.role.RoleChangeException: Can't commit a completed message.

Bug Medium Resolved Done Somashekhar Javalagi Sam Hague csit:3node csit:exception csit:failures Mon, 8 Oct 2018 23:10:28 +0000 Mon, 7 Jan 2019 17:09:40 +0000 Mon, 7 Jan 2019 17:09:40 +0000 Fluorine-SR2 Neon openflowplugin-impl 0 5 <p>looks like we have this here too:<br/> <a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-3node-0cmb-1ctl-2cmp-openstack-queens-upstream-stateful-snat-conntrack-oxygen/86" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-3node-0cmb-1ctl-2cmp-openstack-queens-upstream-stateful-snat-conntrack-oxygen/86</a></p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=Avishnoi" class="user-hover" rel="Avishnoi">Avishnoi</a>, as we continue to dig in to our cluster problems, I'm wondering if we have someone in OFP<br/> that can look at these things as we find them?</p> <p>Looking at logs at <a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-3node-0cmb-1ctl-2cmp-openstack-queens-upstream-stateful-oxygen/95/" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-3node-0cmb-1ctl-2cmp-openstack-queens-upstream-stateful-oxygen/95/</a></p> <p>1. control-1 is repeatedly connecting to 2 of the odls<br/> 2018-10-07T21:49:21.116Z|00045|rconn|INFO|br-int&lt;-&gt;tcp:10.30.170.87:6653: connection closed by peer<br/> 2018-10-07T21:49:21.138Z|00046|rconn|INFO|br-int&lt;-&gt;tcp:10.30.170.108:6653: connection closed by peer</p> <p>2. Control-1 pcap on 6653 show ODL sending echo request/reply and after it goes 5 seconds without sending echo request, it sends a FIN. We see 6 such reconnects to each.<br/> <a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-3node-0cmb-1ctl-2cmp-openstack-queens-upstream-stateful-oxygen/95/control_1/" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-3node-0cmb-1ctl-2cmp-openstack-queens-upstream-stateful-oxygen/95/control_1/</a></p> <p>3. Same pattern repeates on compute-0 but to different controllers and about 56 reconnects each.<br/> 2018-10-07T21:53:07.965Z|00054|rconn|INFO|br-int&lt;-&gt;tcp:10.30.170.156:6653: connection closed by peer<br/> 2018-10-07T21:53:07.969Z|00055|rconn|INFO|br-int&lt;-&gt;tcp:10.30.170.108:6653: connection closed by peer</p> <p>4. It looks like the ODLs that don't become leaders/masters of that device are disconnecting the devices. </p> <p>5. We're seeing repeated role change messages in ODL logs in tune with disconnects. One of them is the cause, other is effect. No idea which is which.</p> <p>In all the cases, I see that whenever the role request is sent to the switch, many of the times this error is coming. Because, though the role request is triggered from the plugin, it is not sending the openflow message for that through channel. Few of the times, I am able to see in wireshark traces that role request is successfully sent and this error is not there.</p> <p> </p> <p>So we have to see why many of the role requests are not being sent by openflowplugin, though triggered.</p> <p>For this I have added debug logs in review : <a href="https://git.opendaylight.org/gerrit/#/c/77372/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/77372/</a>.]</p> <p>I have also triggered test-openflowplugin-cluster-netvirt csit on this review.</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=shague" class="user-hover" rel="shague">shague</a>, is this  job going to reproduce the scenario? If not, can you please suggest which job to run on this openfllowplugin review?</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=Somashekhar" class="user-hover" rel="Somashekhar">Somashekhar</a>, yeah the patch test you ran has this "Was not able to propagate SLAVE" in it's <a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-3node-0cmb-1ctl-2cmp-openstack-queens-gate-stateful-neon/8/odl_1/odl1_karaf.log.gz" class="external-link" target="_blank" rel="nofollow noopener">karaf.log</a></p> <p>i think we can find a faster job maybe if that helps, but it's there. did your debugs work?</p> <p>Apart from the first time, every time the slave request proposal to the switch is failing, which tries to get the generation id by sending a dummy role. The request may be failing because of not resetting the old entry in the outbound queue. Which is wrongly marked as completed task. I will add some changes to reset the outbound queue before the role request is sent and will run the job on same.</p> <p>That has been fixed now. I restarted the csit on the debug patch. There was an infra issue where the vm images were deleted and that has been fixed since.</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=Avishnoi" class="user-hover" rel="Avishnoi">Avishnoi</a>, after adding switch idle handler registration at the end of the tcp channel initialization, I am not able to see this issue. The slave role is being set properly.</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=shague" class="user-hover" rel="shague">shague</a>, Can you please confirm if the issue is there with this change or not, by running csit job</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=jluhrsen" class="user-hover" rel="jluhrsen">jluhrsen</a> / <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=shague" class="user-hover" rel="shague">shague</a> , Could you please confirm if the error has been removed with the review.</p> <p><a href="https://git.opendaylight.org/gerrit/#/c/78199/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/78199/</a></p> <p>The infra is down again so we can't stack the jobs. We need to fix that so that we can run the tests for this patch to verify the fix.</p> <p>I am fine with merging the patch if it does pass the ofp tests and looks to solve the problem.</p> <p>Is the issue resolved? We wanted to ensure before merging the patch.</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=Arunprakash" class="user-hover" rel="Arunprakash">Arunprakash</a>, <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=shague" class="user-hover" rel="shague">shague</a>, so the issue of default flows not being there seems to still<br/> happen from time to time. <a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-3node-0cmb-1ctl-2cmp-openstack-queens-upstream-stateful-oxygen/166/robot-plugin/log_full.html" class="external-link" target="_blank" rel="nofollow noopener">Example here</a>, but even<br/> when that is not the failure in our jobs, we still see this WARN messages a lot:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> org.opendaylight.openflowplugin.impl - 0.6.5 | Not able to set SLAVE role on device openflow:102142247889978, reason: Was not able to propagate SLAVE role on device. Error: org.opendaylight.openflowplugin.impl.role.RoleChangeException: Can't commit a completed message. </pre> </div></div> <p>so, potentially the WARN message is not even related to the default flows missing.</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=jluhrsen" class="user-hover" rel="jluhrsen">jluhrsen</a> we are debugging 2 issues here, one is the exception tracked with this jira. "Not able to set SLAVE role on device"</p> <p>For this, we have raised a patch "<a href="https://git.opendaylight.org/gerrit/#/c/78199/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/78199/</a>" and wanted to run CSIT for the patch and confirm if the error is no longer seen before merging the review.</p> <p>Another one is default flows missing, which is tracked by <a href="https://jira.opendaylight.org/browse/OPNFLWPLUG-1056" title="Default tables missing" class="issue-link" data-issue-key="OPNFLWPLUG-1056"><del>OPNFLWPLUG-1056</del></a>. Even we have raised a debug patch <a href="https://git.opendaylight.org/gerrit/#/c/78730/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/78730/</a> to get more logs, we need to run csit on this and collect some additional logs.</p> <p> </p> <p> </p> <blockquote> <p>Jamo Luhrsen we are debugging 2 issues here, one is the exception tracked with this jira. "Not able to set SLAVE role on device"</p> <p>For this, we have raised a patch "https://git.opendaylight.org/gerrit/#/c/78199/" and wanted to run CSIT for the patch and confirm if the error is no longer seen before merging the review.</p></blockquote> <p>Ok, it looks like you already used our patch test to run CSIT. This is the <a href="https://jenkins.opendaylight.org/releng/job/netvirt-csit-3node-0cmb-1ctl-2cmp-openstack-queens-gate-stateful-neon/19/" class="external-link" target="_blank" rel="nofollow noopener">job you triggered</a><br/> when you gave that 'test-openflowplugin-cluster-netvirt' back on 11/27. I don't see this<br/> WARN anymore in those three karaf logs <a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-3node-0cmb-1ctl-2cmp-openstack-queens-gate-stateful-neon/19/odl_1/odl1_karaf.log.gz" class="external-link" target="_blank" rel="nofollow noopener">1</a>[2|https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-3node-0cmb-1ctl-2cmp-openstack-queens-gate-stateful-neon/19/odl_2/odl2_karaf.log.gz][3|https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-3node-0cmb-1ctl-2cmp-openstack-queens-gate-stateful-neon/19/odl_3/odl3_karaf.log.gz]</p> <p>I put a +1 on that patch because of this. Feel free to run patch-test again if<br/> you think it's needed since that patch has sat for almost a month now.</p> <blockquote> <p>Another one is default flows missing, which is tracked by <a href="https://jira.opendaylight.org/browse/OPNFLWPLUG-1056" title="Default tables missing" class="issue-link" data-issue-key="OPNFLWPLUG-1056"><del>OPNFLWPLUG-1056</del></a>. Even we have raised a debug patch <a href="https://git.opendaylight.org/gerrit/#/c/78730/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/78730/</a> to get more logs, we need to run csit on this and collect some additional logs.</p></blockquote> <p>ok, let's let this 1039 be just for the WARN message and looks like that might<br/> be resolved soon then. I'll go comment on 1056 now.</p> <p>Thanks <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=Arunprakash" class="user-hover" rel="Arunprakash">Arunprakash</a></p> Duplicate OPNFLWPLUG-1035 Development Epic Link NETVIRT-996 Rank 0|i03jfr: