OpenDaylight JIRA

OpenDaylight JIRA https://jira.opendaylight.org This file is an XML representation of an issue en-us

8.20.10

820010

22-06-2022

[OPNFLWPLUG-939] Regression in Switch scalability test (carbon) https://jira.opendaylight.org/browse/OPNFLWPLUG-939 OpenFlowPlugin <p>It seems we can hardly do more than 100 switches in carbon/nitrogen:</p> <p><a href="https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-1node-periodic-sw-scalability-daily-only-carbon/plot/Switch%20Scalability/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-1node-periodic-sw-scalability-daily-only-carbon/plot/Switch%20Scalability/</a></p> <p><a href="https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-1node-periodic-sw-scalability-daily-only-nitrogen/plot/Switch%20Scalability/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-1node-periodic-sw-scalability-daily-only-nitrogen/plot/Switch%20Scalability/</a></p> <p>However boron seems pretty good:</p> <p><a href="https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-1node-periodic-sw-scalability-daily-only-boron/plot/Switch%20Scalability/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-1node-periodic-sw-scalability-daily-only-boron/plot/Switch%20Scalability/</a></p> <p>Operating System: All<br/> Platform: All</p> OPNFLWPLUG-939

Regression in Switch scalability test (carbon)

Bug Resolved Done Luis Gomez Luis Gomez Mon, 4 Sep 2017 01:04:13 +0000 Mon, 27 Sep 2021 09:02:05 +0000 Wed, 27 Sep 2017 15:15:11 +0000 General 0 1 <p>From the karaf log, it seems switches fail to connect:</p> <p>2017-09-03 11:14:42,190 | ERROR | pool-43-thread-1 | OutboundQueueProviderImpl | 278 - org.opendaylight.openflowplugin.impl - 0.5.0.SNAPSHOT | No queue present, failing request<br/> 2017-09-03 11:14:42,195 | ERROR | pool-43-thread-1 | OutboundQueueProviderImpl | 278 - org.opendaylight.openflowplugin.impl - 0.5.0.SNAPSHOT | No queue present, failing request<br/> 2017-09-03 11:14:42,195 | ERROR | pool-43-thread-1 | OutboundQueueProviderImpl | 278 - org.opendaylight.openflowplugin.impl - 0.5.0.SNAPSHOT | No queue present, failing request</p> <p>Here is the karaf log with INFO level:</p> <p><a href="https://logs.opendaylight.org/releng/jenkins092/openflowplugin-csit-1node-periodic-sw-scalability-daily-only-nitrogen/129/odl1_karaf.log.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/openflowplugin-csit-1node-periodic-sw-scalability-daily-only-nitrogen/129/odl1_karaf.log.gz</a></p> <p>It seems like in second iteration of 200 switches, many of the OF connection remain slave (never move to master).</p> <p>Can you try it with this patch: <a href="https://git.opendaylight.org/gerrit/#/c/62607/8" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/62607/8</a> Luis?</p> <p>Sure, I just triggered the build. It will take some time to start given existing jenkins queue.</p> <p>The result from yesterday run with the patch does not show better:</p> <p><a href="https://logs.opendaylight.org/releng/jenkins092/openflowplugin-csit-1node-periodic-sw-scalability-daily-only-nitrogen/133/odl1_karaf.log.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/openflowplugin-csit-1node-periodic-sw-scalability-daily-only-nitrogen/133/odl1_karaf.log.gz</a></p> <p>OK, after looking at the karaf log in detail, this is why scalability test fails and controller breaks. See below scenarios:</p> <p>1) In normal scenario with few switches the device is processed immediately after it connects:</p> <p>2017-09-13 18:51:47,192 | INFO | entLoopGroup-7-8 | ContextChainHolderImpl | 278 - org.opendaylight.openflowplugin.impl - 0.5.0 | Device openflow:1 connected.<br/> 2017-09-13 18:51:47,192 | INFO | entLoopGroup-7-8 | ContextChainHolderImpl | 278 - org.opendaylight.openflowplugin.impl - 0.5.0 | No context chain found for device: openflow:1, creating new.<br/> 2017-09-13 18:51:47,192 | INFO | entLoopGroup-7-8 | DeviceManagerImpl | 278 - org.opendaylight.openflowplugin.impl - 0.5.0 | ConnectionEvent: Device connected to controller, Device:/192.168.0.24:49443, NodeId:Uri <span class="error">[_value=openflow:1]</span><br/> 2017-09-13 18:51:47,192 | INFO | entLoopGroup-7-8 | RoleContextImpl | 278 - org.opendaylight.openflowplugin.impl - 0.5.0 | Started timer for setting SLAVE role on device openflow:1 if no role will be set in 20s.</p> <p>2017-09-13 18:51:47,200 | INFO | lt-dispatcher-18 | GuardedContextImpl | 278 - org.opendaylight.openflowplugin.impl - 0.5.0 | Starting DeviceContextImpl<span class="error">[NEW]</span> service for node openflow:1<br/> 2017-09-13 18:51:47,202 | INFO | entLoopGroup-7-8 | DeviceInitializationUtil | 278 - org.opendaylight.openflowplugin.impl - 0.5.0 | IP address of the node openflow:1 is: IpAddress [_ipv4Address=Ipv4Address <span class="error">[_value=192.168.0.24]</span>]<br/> 2017-09-13 18:51:47,202 | INFO | entLoopGroup-7-8 | DeviceInitializationUtil | 278 - org.opendaylight.openflowplugin.impl - 0.5.0 | Port number of the node openflow:1 is: 49443<br/> 2017-09-13 18:51:47,208 | INFO | entLoopGroup-7-8 | OF13DeviceInitializer | 278 - org.opendaylight.openflowplugin.impl - 0.5.0 | Static node openflow:1 info: OFPMPMETERFEATURES collected<br/> 2017-09-13 18:51:47,208 | INFO | entLoopGroup-7-8 | OF13DeviceInitializer | 278 - org.opendaylight.openflowplugin.impl - 0.5.0 | Static node openflow:1 info: OFPMPGROUPFEATURES collected<br/> 2017-09-13 18:51:47,211 | INFO | entLoopGroup-7-8 | OF13DeviceInitializer | 278 - org.opendaylight.openflowplugin.impl - 0.5.0 | Static node openflow:1 info: OFPMPPORTDESC collected<br/> 2017-09-13 18:51:47,211 | INFO | entLoopGroup-7-8 | OF13DeviceInitializer | 278 - org.opendaylight.openflowplugin.impl - 0.5.0 | Static node openflow:1 successfully finished collecting<br/> 2017-09-13 18:51:47,211 | INFO | lt-dispatcher-18 | GuardedContextImpl | 278 - org.opendaylight.openflowplugin.impl - 0.5.0 | Starting RpcContextImpl<span class="error">[NEW]</span> service for node openflow:1<br/> 2017-09-13 18:51:47,215 | INFO | lt-dispatcher-18 | GuardedContextImpl | 278 - org.opendaylight.openflowplugin.impl - 0.5.0 | Starting StatisticsContextImpl<span class="error">[NEW]</span> service for node openflow:1<br/> 2017-09-13 18:51:47,215 | INFO | lt-dispatcher-18 | GuardedContextImpl | 278 - org.opendaylight.openflowplugin.impl - 0.5.0 | Starting RoleContextImpl<span class="error">[NEW]</span> service for node openflow:1<br/> 2017-09-13 18:51:47,216 | INFO | lt-dispatcher-18 | SalRoleServiceImpl | 278 - org.opendaylight.openflowplugin.impl - 0.5.0 | SetRole called with input:SetRoleInput [_controllerRole=BECOMEMASTER, _node=NodeRef [_value=KeyedInstanceIdentifier</p> {targetType=interface org.opendaylight.yang.gen.v1.urn.opendaylight.inventory.rev130819.nodes.Node, path=[org.opendaylight.yang.gen.v1.urn.opendaylight.inventory.rev130819.Nodes, org.opendaylight.yang.gen.v1.urn.opendaylight.inventory.rev130819.nodes.Node[key=NodeKey [_id=Uri [_value=openflow:1]]]]} <p>], augmentation=[]]<br/> 2017-09-13 18:51:47,224 | INFO | lt-dispatcher-18 | SalRoleServiceImpl | 278 - org.opendaylight.openflowplugin.impl - 0.5.0 | RoleChangeTask called on device:openflow:1 OFPRole:BECOMEMASTER<br/> 2017-09-13 18:51:47,224 | INFO | lt-dispatcher-18 | RoleService | 278 - org.opendaylight.openflowplugin.impl - 0.5.0 | getGenerationIdFromDevice called for device: openflow:1<br/> 2017-09-13 18:51:47,224 | INFO | lt-dispatcher-18 | ContextChainImpl | 278 - org.opendaylight.openflowplugin.impl - 0.5.0 | Started clustering services for node openflow:1<br/> 2017-09-13 18:51:47,225 | INFO | entLoopGroup-7-8 | RoleService | 278 - org.opendaylight.openflowplugin.impl - 0.5.0 | submitRoleChange called for device:Uri <span class="error">[_value=openflow:1]</span>, role:BECOMEMASTER<br/> 2017-09-13 18:51:47,226 | INFO | entLoopGroup-7-8 | RoleService | 278 - org.opendaylight.openflowplugin.impl - 0.5.0 | submitRoleChange onSuccess for device:Uri <span class="error">[_value=openflow:1]</span>, role:BECOMEMASTER<br/> 2017-09-13 18:51:47,226 | INFO | entLoopGroup-7-8 | FlowNodeReconciliationImpl | 268 - org.opendaylight.openflowplugin.applications.forwardingrules-manager - 0.5.0 | Triggering reconciliation for device NodeKey [_id=Uri <span class="error">[_value=openflow:1]</span>]<br/> 2017-09-13 18:51:47,226 | INFO | entLoopGroup-7-8 | ContextChainHolderImpl | 278 - org.opendaylight.openflowplugin.impl - 0.5.0 | Device openflow:1 connection is enabled by reconciliation framework.<br/> 2017-09-13 18:51:47,229 | INFO | entLoopGroup-7-8 | StatisticsManagerImpl | 278 - org.opendaylight.openflowplugin.impl - 0.5.0 | Scheduling statistics poll for device: Uri <span class="error">[_value=openflow:1]</span><br/> 2017-09-13 18:51:47,229 | INFO | entLoopGroup-7-8 | ContextChainImpl | 278 - org.opendaylight.openflowplugin.impl - 0.5.0 | Device openflow:1 is able to work as master.<br/> 2017-09-13 18:51:47,229 | INFO | entLoopGroup-7-8 | DeviceManagerImpl | 278 - org.opendaylight.openflowplugin.impl - 0.5.0 | Publishing node added notification for Uri <span class="error">[_value=openflow:1]</span></p> <p>2) When more switches start to connect simultaneously, controller takes more time to process the switches and after 200 switches it is very likely this time overpasses the 20 secs max time to set up slave, so in this case there is first a slave role set and then a master role set after switch is served:</p> <p>2017-09-13 17:36:47,384 | INFO | entLoopGroup-7-3 | ContextChainHolderImpl | 278 - org.opendaylight.openflowplugin.impl - 0.5.0 | Device openflow:99 connected.<br/> 2017-09-13 17:36:47,384 | INFO | entLoopGroup-7-3 | ContextChainHolderImpl | 278 - org.opendaylight.openflowplugin.impl - 0.5.0 | No context chain found for device: openflow:99, creating new.<br/> 2017-09-13 17:36:47,384 | INFO | entLoopGroup-7-3 | DeviceManagerImpl | 278 - org.opendaylight.openflowplugin.impl - 0.5.0 | ConnectionEvent: Device connected to controller, Device:/192.168.0.24:49062, NodeId:Uri <span class="error">[_value=openflow:99]</span><br/> 2017-09-13 17:36:47,384 | INFO | entLoopGroup-7-3 | RoleContextImpl | 278 - org.opendaylight.openflowplugin.impl - 0.5.0 | Started timer for setting SLAVE role on device openflow:99 if no role will be set in 20s.</p> <p>2017-09-13 17:37:07,394 | INFO | pool-40-thread-1 | SalRoleServiceImpl | 278 - org.opendaylight.openflowplugin.impl - 0.5.0 | SetRole called with input:SetRoleInput [_controllerRole=BECOMESLAVE, _node=NodeRef [_value=KeyedInstanceIdentifier</p> {targetType=interface org.opendaylight.yang.gen.v1.urn.opendaylight.inventory.rev130819.nodes.Node, path=[org.opendaylight.yang.gen.v1.urn.opendaylight.inventory.rev130819.Nodes, org.opendaylight.yang.gen.v1.urn.opendaylight.inventory.rev130819.nodes.Node[key=NodeKey [_id=Uri [_value=openflow:99]]]]} <p>], augmentation=[]]<br/> 2017-09-13 17:37:07,394 | INFO | pool-40-thread-1 | SalRoleServiceImpl | 278 - org.opendaylight.openflowplugin.impl - 0.5.0 | RoleChangeTask called on device:openflow:99 OFPRole:BECOMESLAVE<br/> 2017-09-13 17:37:07,394 | INFO | pool-40-thread-1 | RoleService | 278 - org.opendaylight.openflowplugin.impl - 0.5.0 | getGenerationIdFromDevice called for device: openflow:99<br/> 2017-09-13 17:37:07,899 | INFO | entLoopGroup-7-3 | RoleService | 278 - org.opendaylight.openflowplugin.impl - 0.5.0 | submitRoleChange called for device:Uri <span class="error">[_value=openflow:99]</span>, role:BECOMESLAVE<br/> 2017-09-13 17:37:08,398 | INFO | entLoopGroup-7-3 | RoleService | 278 - org.opendaylight.openflowplugin.impl - 0.5.0 | submitRoleChange onSuccess for device:Uri <span class="error">[_value=openflow:99]</span>, role:BECOMESLAVE<br/> 2017-09-13 17:37:08,398 | INFO | entLoopGroup-7-3 | ReconciliationManagerImpl | 271 - org.opendaylight.openflowplugin.applications.reconciliation-framework - 0.5.0 | Stopping reconciliation for node Uri <span class="error">[_value=openflow:99]</span><br/> 2017-09-13 17:37:08,398 | INFO | entLoopGroup-7-3 | ContextChainHolderImpl | 278 - org.opendaylight.openflowplugin.impl - 0.5.0 | Role SLAVE was granted to device openflow:99</p> <p>2017-09-13 17:38:44,241 | INFO | lt-dispatcher-19 | GuardedContextImpl | 278 - org.opendaylight.openflowplugin.impl - 0.5.0 | Starting DeviceContextImpl<span class="error">[NEW]</span> service for node openflow:99<br/> 2017-09-13 17:38:44,250 | INFO | entLoopGroup-7-3 | DeviceInitializationUtil | 278 - org.opendaylight.openflowplugin.impl - 0.5.0 | IP address of the node openflow:99 is: IpAddress [_ipv4Address=Ipv4Address <span class="error">[_value=192.168.0.24]</span>]<br/> 2017-09-13 17:38:44,250 | INFO | entLoopGroup-7-3 | DeviceInitializationUtil | 278 - org.opendaylight.openflowplugin.impl - 0.5.0 | Port number of the node openflow:99 is: 49062<br/> 2017-09-13 17:38:44,263 | INFO | entLoopGroup-7-3 | OF13DeviceInitializer | 278 - org.opendaylight.openflowplugin.impl - 0.5.0 | Static node openflow:99 info: OFPMPMETERFEATURES collected<br/> 2017-09-13 17:38:44,263 | INFO | entLoopGroup-7-3 | OF13DeviceInitializer | 278 - org.opendaylight.openflowplugin.impl - 0.5.0 | Static node openflow:99 info: OFPMPGROUPFEATURES collected<br/> 2017-09-13 17:38:44,263 | INFO | entLoopGroup-7-3 | OF13DeviceInitializer | 278 - org.opendaylight.openflowplugin.impl - 0.5.0 | Static node openflow:99 info: OFPMPPORTDESC collected<br/> 2017-09-13 17:38:44,264 | INFO | entLoopGroup-7-3 | OF13DeviceInitializer | 278 - org.opendaylight.openflowplugin.impl - 0.5.0 | Static node openflow:99 successfully finished collecting<br/> 2017-09-13 17:38:44,264 | INFO | lt-dispatcher-19 | GuardedContextImpl | 278 - org.opendaylight.openflowplugin.impl - 0.5.0 | Starting RpcContextImpl<span class="error">[NEW]</span> service for node openflow:99<br/> 2017-09-13 17:38:44,284 | INFO | lt-dispatcher-19 | GuardedContextImpl | 278 - org.opendaylight.openflowplugin.impl - 0.5.0 | Starting StatisticsContextImpl<span class="error">[NEW]</span> service for node openflow:99<br/> 2017-09-13 17:38:44,284 | INFO | lt-dispatcher-19 | GuardedContextImpl | 278 - org.opendaylight.openflowplugin.impl - 0.5.0 | Starting RoleContextImpl<span class="error">[NEW]</span> service for node openflow:99<br/> 2017-09-13 17:38:44,285 | INFO | lt-dispatcher-19 | SalRoleServiceImpl | 278 - org.opendaylight.openflowplugin.impl - 0.5.0 | SetRole called with input:SetRoleInput [_controllerRole=BECOMEMASTER, _node=NodeRef [_value=KeyedInstanceIdentifier</p> {targetType=interface org.opendaylight.yang.gen.v1.urn.opendaylight.inventory.rev130819.nodes.Node, path=[org.opendaylight.yang.gen.v1.urn.opendaylight.inventory.rev130819.Nodes, org.opendaylight.yang.gen.v1.urn.opendaylight.inventory.rev130819.nodes.Node[key=NodeKey [_id=Uri [_value=openflow:99]]]]} <p>], augmentation=[]]<br/> 2017-09-13 17:38:44,285 | INFO | lt-dispatcher-19 | SalRoleServiceImpl | 278 - org.opendaylight.openflowplugin.impl - 0.5.0 | RoleChangeTask called on device:openflow:99 OFPRole:BECOMEMASTER<br/> 2017-09-13 17:38:44,285 | INFO | lt-dispatcher-19 | RoleService | 278 - org.opendaylight.openflowplugin.impl - 0.5.0 | getGenerationIdFromDevice called for device: openflow:99<br/> 2017-09-13 17:38:44,285 | INFO | lt-dispatcher-19 | ContextChainImpl | 278 - org.opendaylight.openflowplugin.impl - 0.5.0 | Started clustering services for node openflow:99<br/> 2017-09-13 17:38:44,292 | INFO | entLoopGroup-7-3 | RoleService | 278 - org.opendaylight.openflowplugin.impl - 0.5.0 | submitRoleChange called for device:Uri <span class="error">[_value=openflow:99]</span>, role:BECOMEMASTER<br/> 2017-09-13 17:38:44,297 | INFO | entLoopGroup-7-3 | RoleService | 278 - org.opendaylight.openflowplugin.impl - 0.5.0 | submitRoleChange onSuccess for device:Uri <span class="error">[_value=openflow:99]</span>, role:BECOMEMASTER<br/> 2017-09-13 17:38:44,381 | INFO | entLoopGroup-7-3 | FlowNodeReconciliationImpl | 268 - org.opendaylight.openflowplugin.applications.forwardingrules-manager - 0.5.0 | Triggering reconciliation for device NodeKey [_id=Uri <span class="error">[_value=openflow:99]</span>]<br/> 2017-09-13 17:38:44,383 | INFO | dapter-thread-17 | ContextChainHolderImpl | 278 - org.opendaylight.openflowplugin.impl - 0.5.0 | Device openflow:99 connection is enabled by reconciliation framework.<br/> 2017-09-13 17:38:44,461 | INFO | dapter-thread-17 | StatisticsManagerImpl | 278 - org.opendaylight.openflowplugin.impl - 0.5.0 | Scheduling statistics poll for device: Uri <span class="error">[_value=openflow:99]</span><br/> 2017-09-13 17:38:44,461 | INFO | dapter-thread-17 | ContextChainImpl | 278 - org.opendaylight.openflowplugin.impl - 0.5.0 | Device openflow:99 is able to work as master.<br/> 2017-09-13 17:38:44,461 | INFO | dapter-thread-17 | DeviceManagerImpl | 278 - org.opendaylight.openflowplugin.impl - 0.5.0 | Publishing node added notification for Uri <span class="error">[_value=openflow:99]</span></p> <p>3) Finally, this is the problematic scenario that happens when a switch is processed just before the 20 secs max time to set up slave, controller in this case mess ups all as it tries to setup master and slave roles simultaneously as you can see below:</p> <p>2017-09-13 17:36:47,413 | INFO | entLoopGroup-7-6 | ContextChainHolderImpl | 278 - org.opendaylight.openflowplugin.impl - 0.5.0 | Device openflow:100 connected.<br/> 2017-09-13 17:36:47,413 | INFO | entLoopGroup-7-6 | ContextChainHolderImpl | 278 - org.opendaylight.openflowplugin.impl - 0.5.0 | No context chain found for device: openflow:100, creating new.<br/> 2017-09-13 17:36:47,413 | INFO | entLoopGroup-7-6 | DeviceManagerImpl | 278 - org.opendaylight.openflowplugin.impl - 0.5.0 | ConnectionEvent: Device connected to controller, Device:/192.168.0.24:49257, NodeId:Uri <span class="error">[_value=openflow:100]</span><br/> 2017-09-13 17:36:47,414 | INFO | entLoopGroup-7-6 | RoleContextImpl | 278 - org.opendaylight.openflowplugin.impl - 0.5.0 | Started timer for setting SLAVE role on device openflow:100 if no role will be set in 20s.</p> <p>2017-09-13 17:37:07,285 | INFO | lt-dispatcher-49 | GuardedContextImpl | 278 - org.opendaylight.openflowplugin.impl - 0.5.0 | Starting DeviceContextImpl<span class="error">[NEW]</span> service for node openflow:100<br/> 2017-09-13 17:37:07,294 | INFO | entLoopGroup-7-6 | DeviceInitializationUtil | 278 - org.opendaylight.openflowplugin.impl - 0.5.0 | IP address of the node openflow:100 is: IpAddress [_ipv4Address=Ipv4Address <span class="error">[_value=192.168.0.24]</span>]<br/> 2017-09-13 17:37:07,294 | INFO | entLoopGroup-7-6 | DeviceInitializationUtil | 278 - org.opendaylight.openflowplugin.impl - 0.5.0 | Port number of the node openflow:100 is: 49257<br/> 2017-09-13 17:37:07,304 | INFO | entLoopGroup-7-6 | OF13DeviceInitializer | 278 - org.opendaylight.openflowplugin.impl - 0.5.0 | Static node openflow:100 info: OFPMPMETERFEATURES collected<br/> 2017-09-13 17:37:07,304 | INFO | entLoopGroup-7-6 | OF13DeviceInitializer | 278 - org.opendaylight.openflowplugin.impl - 0.5.0 | Static node openflow:100 info: OFPMPGROUPFEATURES collected<br/> 2017-09-13 17:37:07,305 | INFO | entLoopGroup-7-6 | OF13DeviceInitializer | 278 - org.opendaylight.openflowplugin.impl - 0.5.0 | Static node openflow:100 info: OFPMPPORTDESC collected<br/> 2017-09-13 17:37:07,305 | INFO | entLoopGroup-7-6 | OF13DeviceInitializer | 278 - org.opendaylight.openflowplugin.impl - 0.5.0 | Static node openflow:100 successfully finished collecting<br/> 2017-09-13 17:37:07,305 | INFO | lt-dispatcher-49 | GuardedContextImpl | 278 - org.opendaylight.openflowplugin.impl - 0.5.0 | Starting RpcContextImpl<span class="error">[NEW]</span> service for node openflow:100<br/> 2017-09-13 17:37:07,309 | INFO | lt-dispatcher-49 | GuardedContextImpl | 278 - org.opendaylight.openflowplugin.impl - 0.5.0 | Starting StatisticsContextImpl<span class="error">[NEW]</span> service for node openflow:100<br/> 2017-09-13 17:37:07,309 | INFO | lt-dispatcher-49 | GuardedContextImpl | 278 - org.opendaylight.openflowplugin.impl - 0.5.0 | Starting RoleContextImpl<span class="error">[NEW]</span> service for node openflow:100<br/> 2017-09-13 17:37:07,309 | INFO | lt-dispatcher-49 | SalRoleServiceImpl | 278 - org.opendaylight.openflowplugin.impl - 0.5.0 | SetRole called with input:SetRoleInput [_controllerRole=BECOMEMASTER, _node=NodeRef [_value=KeyedInstanceIdentifier</p> {targetType=interface org.opendaylight.yang.gen.v1.urn.opendaylight.inventory.rev130819.nodes.Node, path=[org.opendaylight.yang.gen.v1.urn.opendaylight.inventory.rev130819.Nodes, org.opendaylight.yang.gen.v1.urn.opendaylight.inventory.rev130819.nodes.Node[key=NodeKey [_id=Uri [_value=openflow:100]]]]} <p>], augmentation=[]]<br/> 2017-09-13 17:37:07,309 | INFO | lt-dispatcher-49 | SalRoleServiceImpl | 278 - org.opendaylight.openflowplugin.impl - 0.5.0 | RoleChangeTask called on device:openflow:100 OFPRole:BECOMEMASTER<br/> 2017-09-13 17:37:07,309 | INFO | lt-dispatcher-49 | RoleService | 278 - org.opendaylight.openflowplugin.impl - 0.5.0 | getGenerationIdFromDevice called for device: openflow:100<br/> 2017-09-13 17:37:07,309 | INFO | lt-dispatcher-49 | ContextChainImpl | 278 - org.opendaylight.openflowplugin.impl - 0.5.0 | Started clustering services for node openflow:100<br/> 2017-09-13 17:37:07,349 | INFO | entLoopGroup-7-6 | RoleService | 278 - org.opendaylight.openflowplugin.impl - 0.5.0 | submitRoleChange called for device:Uri <span class="error">[_value=openflow:100]</span>, role:BECOMEMASTER</p> <p>2017-09-13 17:37:07,422 | INFO | pool-40-thread-1 | SalRoleServiceImpl | 278 - org.opendaylight.openflowplugin.impl - 0.5.0 | SetRole called with input:SetRoleInput [_controllerRole=BECOMESLAVE, _node=NodeRef [_value=KeyedInstanceIdentifier</p> {targetType=interface org.opendaylight.yang.gen.v1.urn.opendaylight.inventory.rev130819.nodes.Node, path=[org.opendaylight.yang.gen.v1.urn.opendaylight.inventory.rev130819.Nodes, org.opendaylight.yang.gen.v1.urn.opendaylight.inventory.rev130819.nodes.Node[key=NodeKey [_id=Uri [_value=openflow:100]]]]} <p>], augmentation=[]]<br/> 2017-09-13 17:37:07,422 | INFO | pool-40-thread-1 | SalRoleServiceImpl | 278 - org.opendaylight.openflowplugin.impl - 0.5.0 | RoleChangeTask called on device:openflow:100 OFPRole:BECOMESLAVE<br/> 2017-09-13 17:37:07,422 | INFO | pool-40-thread-1 | RoleService | 278 - org.opendaylight.openflowplugin.impl - 0.5.0 | getGenerationIdFromDevice called for device: openflow:100<br/> 2017-09-13 17:37:07,422 | WARN | pool-40-thread-1 | ContextChainHolderImpl | 278 - org.opendaylight.openflowplugin.impl - 0.5.0 | Not able to set MASTER role on device openflow:100, reason: Was not able to propagate MASTER role on device. Error: java.util.concurrent.CancellationException: Task was cancelled.<br/> 2017-09-13 17:37:07,422 | WARN | pool-40-thread-1 | ContextChainHolderImpl | 278 - org.opendaylight.openflowplugin.impl - 0.5.0 | This mastering is mandatory, destroying context chain and closing connection for device openflow:100.<br/> 2017-09-13 17:37:07,422 | INFO | pool-40-thread-1 | ReconciliationManagerImpl | 271 - org.opendaylight.openflowplugin.applications.reconciliation-framework - 0.5.0 | Stopping reconciliation for node Uri <span class="error">[_value=openflow:100]</span><br/> 2017-09-13 17:37:07,423 | INFO | pool-40-thread-1 | ContextChainImpl | 278 - org.opendaylight.openflowplugin.impl - 0.5.0 | Closed clustering services registration for node openflow:100<br/> 2017-09-13 17:37:07,423 | INFO | pool-40-thread-1 | GuardedContextImpl | 278 - org.opendaylight.openflowplugin.impl - 0.5.0 | Terminating DeviceContextImpl<span class="error">[RUNNING]</span> service for node openflow:100<br/> 2017-09-13 17:37:07,423 | INFO | pool-40-thread-1 | GuardedContextImpl | 278 - org.opendaylight.openflowplugin.impl - 0.5.0 | Terminating RpcContextImpl<span class="error">[RUNNING]</span> service for node openflow:100<br/> 2017-09-13 17:37:07,446 | INFO | pool-40-thread-1 | GuardedContextImpl | 278 - org.opendaylight.openflowplugin.impl - 0.5.0 | Terminating StatisticsContextImpl<span class="error">[RUNNING]</span> service for node openflow:100<br/> 2017-09-13 17:37:07,446 | INFO | pool-40-thread-1 | StatisticsContextImpl | 278 - org.opendaylight.openflowplugin.impl - 0.5.0 | Stopping running statistics gathering for node openflow:100<br/> 2017-09-13 17:37:07,446 | WARN | pool-40-thread-1 | ContextChainHolderImpl | 278 - org.opendaylight.openflowplugin.impl - 0.5.0 | Not able to set MASTER role on device openflow:100, reason: Initial gathering statistics unsuccessful.<br/> 2017-09-13 17:37:07,446 | WARN | pool-40-thread-1 | ContextChainHolderImpl | 278 - org.opendaylight.openflowplugin.impl - 0.5.0 | This mastering is mandatory, destroying context chain and closing connection for device openflow:100.<br/> 2017-09-13 17:37:07,446 | INFO | pool-40-thread-1 | ReconciliationManagerImpl | 271 - org.opendaylight.openflowplugin.applications.reconciliation-framework - 0.5.0 | Stopping reconciliation for node Uri <span class="error">[_value=openflow:100]</span><br/> 2017-09-13 17:37:07,447 | INFO | pool-40-thread-1 | GuardedContextImpl | 278 - org.opendaylight.openflowplugin.impl - 0.5.0 | Terminating RoleContextImpl<span class="error">[RUNNING]</span> service for node openflow:100<br/> 2017-09-13 17:37:07,451 | WARN | pool-40-thread-1 | ContextChainHolderImpl | 278 - org.opendaylight.openflowplugin.impl - 0.5.0 | Not able to set SLAVE role on device openflow:100, reason: Was not able to propagate SLAVE role on device. Error: org.opendaylight.openflowplugin.impl.role.RoleChangeException: Connection closed.<br/> 2017-09-13 17:37:07,451 | INFO | pool-40-thread-1 | ReconciliationManagerImpl | 271 - org.opendaylight.openflowplugin.applications.reconciliation-framework - 0.5.0 | Stopping reconciliation for node Uri <span class="error">[_value=openflow:100]</span><br/> 2017-09-13 17:37:07,927 | INFO | entLoopGroup-7-6 | RoleService | 278 - org.opendaylight.openflowplugin.impl - 0.5.0 | submitRoleChange onSuccess for device:Uri <span class="error">[_value=openflow:100]</span>, role:BECOMEMASTER</p> <p>Since this can happen with relatively few switches connecting at the same time and the impact is really bad (only solution is restart controller), I am raising this issue as BLOCKER for nitrogen.</p> <p>Can you test it with this patch Luis? <a href="https://git.opendaylight.org/gerrit/#/c/63123/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/63123/</a></p> <p>Result is bit better but I still see the issue:</p> <p><a href="https://logs.opendaylight.org/releng/jenkins092/openflowplugin-csit-1node-periodic-sw-scalability-daily-only-nitrogen/147/odl1_karaf.log.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/openflowplugin-csit-1node-periodic-sw-scalability-daily-only-nitrogen/147/odl1_karaf.log.gz</a></p> <p>— Device disconnects because load —</p> <p>2017-09-14 18:00:19,597 | INFO | entLoopGroup-7-1 | SystemNotificationsListenerImpl | 278 - org.opendaylight.openflowplugin.impl - 0.5.0.SNAPSHOT | ConnectionEvent: Connection closed by device, Device:/10.29.13.226:53000, NodeId:openflow:109<br/> 2017-09-14 18:00:19,597 | INFO | entLoopGroup-7-1 | ContextChainHolderImpl | 278 - org.opendaylight.openflowplugin.impl - 0.5.0.SNAPSHOT | Device openflow:109 disconnected.<br/> 2017-09-14 18:00:19,597 | INFO | entLoopGroup-7-1 | ReconciliationManagerImpl | 271 - org.opendaylight.openflowplugin.applications.reconciliation-framework - 0.5.0.SNAPSHOT | Stopping reconciliation for node Uri <span class="error">[_value=openflow:109]</span><br/> 2017-09-14 18:00:19,597 | INFO | entLoopGroup-7-1 | DeviceManagerImpl | 278 - org.opendaylight.openflowplugin.impl - 0.5.0.SNAPSHOT | Publishing node removed notification for Uri <span class="error">[_value=openflow:109]</span><br/> 2017-09-14 18:00:19,597 | INFO | entLoopGroup-7-1 | ContextChainImpl | 278 - org.opendaylight.openflowplugin.impl - 0.5.0.SNAPSHOT | Closed clustering services registration for node openflow:109<br/> 2017-09-14 18:00:19,597 | INFO | entLoopGroup-7-1 | GuardedContextImpl | 278 - org.opendaylight.openflowplugin.impl - 0.5.0.SNAPSHOT | Terminating DeviceContextImpl<span class="error">[RUNNING]</span> service for node openflow:109<br/> 2017-09-14 18:00:19,597 | INFO | entLoopGroup-7-1 | GuardedContextImpl | 278 - org.opendaylight.openflowplugin.impl - 0.5.0.SNAPSHOT | Terminating RpcContextImpl<span class="error">[RUNNING]</span> service for node openflow:109<br/> 2017-09-14 18:00:19,598 | INFO | entLoopGroup-7-1 | GuardedContextImpl | 278 - org.opendaylight.openflowplugin.impl - 0.5.0.SNAPSHOT | Terminating StatisticsContextImpl<span class="error">[RUNNING]</span> service for node openflow:109<br/> 2017-09-14 18:00:19,598 | INFO | entLoopGroup-7-1 | StatisticsContextImpl | 278 - org.opendaylight.openflowplugin.impl - 0.5.0.SNAPSHOT | Stopping running statistics gathering for node openflow:109<br/> 2017-09-14 18:00:19,598 | INFO | entLoopGroup-7-1 | StatisticsManagerImpl | 278 - org.opendaylight.openflowplugin.impl - 0.5.0.SNAPSHOT | Statistics gathering for device openflow:109 was cancelled.<br/> 2017-09-14 18:00:19,598 | INFO | entLoopGroup-7-1 | GuardedContextImpl | 278 - org.opendaylight.openflowplugin.impl - 0.5.0.SNAPSHOT | Terminating RoleContextImpl<span class="error">[RUNNING]</span> service for node openflow:109<br/> 2017-09-14 18:00:19,598 | INFO | pool-41-thread-1 | DeviceMastershipManager | 268 - org.opendaylight.openflowplugin.applications.forwardingrules-manager - 0.5.0.SNAPSHOT | Unregistered FRM cluster singleton service for service id : openflow:109<br/> 2017-09-14 18:00:21,381 | INFO | lt-dispatcher-21 | ReconciliationManagerImpl | 271 - org.opendaylight.openflowplugin.applications.reconciliation-framework - 0.5.0.SNAPSHOT | Stopping reconciliation for node Uri <span class="error">[_value=openflow:109]</span><br/> 2017-09-14 18:00:21,381 | INFO | lt-dispatcher-21 | ContextChainHolderImpl | 278 - org.opendaylight.openflowplugin.impl - 0.5.0.SNAPSHOT | Role SLAVE was granted to device openflow:109<br/> 2017-09-14 18:00:21,381 | INFO | ofppool-31 | ContextChainImpl | 278 - org.opendaylight.openflowplugin.impl - 0.5.0.SNAPSHOT | Closed clustering services for node openflow:109<br/> 2017-09-14 18:00:21,391 | INFO | lt-dispatcher-24 | ContextChainHolderImpl | 278 - org.opendaylight.openflowplugin.impl - 0.5.0.SNAPSHOT | Try to remove device Uri <span class="error">[_value=openflow:109]</span> from operational DS<br/> 2017-09-14 18:00:21,391 | INFO | lt-dispatcher-24 | ContextChainHolderImpl | 278 - org.opendaylight.openflowplugin.impl - 0.5.0.SNAPSHOT | Removing device from operational DS Uri <span class="error">[_value=openflow:109]</span> was successful</p> <p>— Device Reconnects —</p> <p>2017-09-14 18:00:53,573 | INFO | entLoopGroup-7-8 | ContextChainHolderImpl | 278 - org.opendaylight.openflowplugin.impl - 0.5.0.SNAPSHOT | Device openflow:109 connected.<br/> 2017-09-14 18:00:53,573 | INFO | entLoopGroup-7-8 | ContextChainHolderImpl | 278 - org.opendaylight.openflowplugin.impl - 0.5.0.SNAPSHOT | No context chain found for device: openflow:109, creating new.<br/> 2017-09-14 18:00:53,573 | INFO | entLoopGroup-7-8 | DeviceManagerImpl | 278 - org.opendaylight.openflowplugin.impl - 0.5.0.SNAPSHOT | ConnectionEvent: Device connected to controller, Device:/10.29.13.226:53478, NodeId:Uri <span class="error">[_value=openflow:109]</span><br/> 2017-09-14 18:00:53,573 | INFO | entLoopGroup-7-8 | RoleContextImpl | 278 - org.opendaylight.openflowplugin.impl - 0.5.0.SNAPSHOT | Started timer for setting SLAVE role on device openflow:109 if no role will be set in 20s.</p> <p>— Master process starts —</p> <p>2017-09-14 18:01:12,485 | INFO | lt-dispatcher-24 | GuardedContextImpl | 278 - org.opendaylight.openflowplugin.impl - 0.5.0.SNAPSHOT | Starting DeviceContextImpl<span class="error">[NEW]</span> service for node openflow:109<br/> 2017-09-14 18:01:12,589 | INFO | entLoopGroup-7-8 | DeviceInitializationUtil | 278 - org.opendaylight.openflowplugin.impl - 0.5.0.SNAPSHOT | IP address of the node openflow:109 is: IpAddress [_ipv4Address=Ipv4Address <span class="error">[_value=10.29.13.226]</span>]<br/> 2017-09-14 18:01:12,589 | INFO | entLoopGroup-7-8 | DeviceInitializationUtil | 278 - org.opendaylight.openflowplugin.impl - 0.5.0.SNAPSHOT | Port number of the node openflow:109 is: 53478</p> <p>— 20 secs slave role kicks in the middle and problem happens —</p> <p>2017-09-14 18:01:13,581 | INFO | pool-44-thread-1 | SalRoleServiceImpl | 278 - org.opendaylight.openflowplugin.impl - 0.5.0.SNAPSHOT | SetRole called with input:SetRoleInput [_controllerRole=BECOMESLAVE, _node=NodeRef [_value=KeyedInstanceIdentifier</p> {targetType=interface org.opendaylight.yang.gen.v1.urn.opendaylight.inventory.rev130819.nodes.Node, path=[org.opendaylight.yang.gen.v1.urn.opendaylight.inventory.rev130819.Nodes, org.opendaylight.yang.gen.v1.urn.opendaylight.inventory.rev130819.nodes.Node[key=NodeKey [_id=Uri [_value=openflow:109]]]]} <p>], augmentation=[]]<br/> 2017-09-14 18:01:13,581 | INFO | pool-44-thread-1 | SalRoleServiceImpl | 278 - org.opendaylight.openflowplugin.impl - 0.5.0.SNAPSHOT | RoleChangeTask called on device:openflow:109 OFPRole:BECOMESLAVE<br/> 2017-09-14 18:01:13,581 | INFO | pool-44-thread-1 | RoleService | 278 - org.opendaylight.openflowplugin.impl - 0.5.0.SNAPSHOT | getGenerationIdFromDevice called for device: openflow:109<br/> 2017-09-14 18:01:13,655 | INFO | entLoopGroup-7-8 | OF13DeviceInitializer | 278 - org.opendaylight.openflowplugin.impl - 0.5.0.SNAPSHOT | Static node openflow:109 info: OFPMPMETERFEATURES collected<br/> 2017-09-14 18:01:13,655 | INFO | entLoopGroup-7-8 | OF13DeviceInitializer | 278 - org.opendaylight.openflowplugin.impl - 0.5.0.SNAPSHOT | Static node openflow:109 info: OFPMPGROUPFEATURES collected<br/> 2017-09-14 18:01:13,655 | INFO | entLoopGroup-7-8 | OF13DeviceInitializer | 278 - org.opendaylight.openflowplugin.impl - 0.5.0.SNAPSHOT | Static node openflow:109 info: OFPMPPORTDESC collected<br/> 2017-09-14 18:01:13,656 | INFO | entLoopGroup-7-8 | OF13DeviceInitializer | 278 - org.opendaylight.openflowplugin.impl - 0.5.0.SNAPSHOT | Static node openflow:109 successfully finished collecting<br/> 2017-09-14 18:01:13,656 | INFO | entLoopGroup-7-8 | RoleService | 278 - org.opendaylight.openflowplugin.impl - 0.5.0.SNAPSHOT | submitRoleChange called for device:Uri <span class="error">[_value=openflow:109]</span>, role:BECOMESLAVE<br/> 2017-09-14 18:01:13,658 | INFO | lt-dispatcher-24 | GuardedContextImpl | 278 - org.opendaylight.openflowplugin.impl - 0.5.0.SNAPSHOT | Starting RpcContextImpl<span class="error">[NEW]</span> service for node openflow:109<br/> 2017-09-14 18:01:13,671 | INFO | lt-dispatcher-24 | GuardedContextImpl | 278 - org.opendaylight.openflowplugin.impl - 0.5.0.SNAPSHOT | Starting StatisticsContextImpl<span class="error">[NEW]</span> service for node openflow:109<br/> 2017-09-14 18:01:13,671 | INFO | lt-dispatcher-24 | GuardedContextImpl | 278 - org.opendaylight.openflowplugin.impl - 0.5.0.SNAPSHOT | Starting RoleContextImpl<span class="error">[NEW]</span> service for node openflow:109<br/> 2017-09-14 18:01:13,671 | INFO | lt-dispatcher-24 | SalRoleServiceImpl | 278 - org.opendaylight.openflowplugin.impl - 0.5.0.SNAPSHOT | SetRole called with input:SetRoleInput [_controllerRole=BECOMEMASTER, _node=NodeRef [_value=KeyedInstanceIdentifier</p> {targetType=interface org.opendaylight.yang.gen.v1.urn.opendaylight.inventory.rev130819.nodes.Node, path=[org.opendaylight.yang.gen.v1.urn.opendaylight.inventory.rev130819.Nodes, org.opendaylight.yang.gen.v1.urn.opendaylight.inventory.rev130819.nodes.Node[key=NodeKey [_id=Uri [_value=openflow:109]]]]} <p>], augmentation=[]]<br/> 2017-09-14 18:01:13,671 | INFO | lt-dispatcher-24 | SalRoleServiceImpl | 278 - org.opendaylight.openflowplugin.impl - 0.5.0.SNAPSHOT | RoleChangeTask called on device:openflow:109 OFPRole:BECOMEMASTER<br/> 2017-09-14 18:01:13,671 | INFO | lt-dispatcher-24 | RoleService | 278 - org.opendaylight.openflowplugin.impl - 0.5.0.SNAPSHOT | getGenerationIdFromDevice called for device: openflow:109<br/> 2017-09-14 18:01:13,671 | WARN | lt-dispatcher-24 | ContextChainHolderImpl | 278 - org.opendaylight.openflowplugin.impl - 0.5.0.SNAPSHOT | Not able to set SLAVE role on device openflow:109, reason: Was not able to propagate SLAVE role on device. Error: java.util.concurrent.CancellationException: Task was cancelled.<br/> 2017-09-14 18:01:13,672 | INFO | lt-dispatcher-24 | ReconciliationManagerImpl | 271 - org.opendaylight.openflowplugin.applications.reconciliation-framework - 0.5.0.SNAPSHOT | Stopping reconciliation for node Uri <span class="error">[_value=openflow:109]</span><br/> 2017-09-14 18:01:13,672 | INFO | lt-dispatcher-24 | ContextChainImpl | 278 - org.opendaylight.openflowplugin.impl - 0.5.0.SNAPSHOT | Closed clustering services registration for node openflow:109<br/> 2017-09-14 18:01:13,672 | INFO | lt-dispatcher-24 | GuardedContextImpl | 278 - org.opendaylight.openflowplugin.impl - 0.5.0.SNAPSHOT | Terminating DeviceContextImpl<span class="error">[RUNNING]</span> service for node openflow:109<br/> 2017-09-14 18:01:13,673 | INFO | lt-dispatcher-24 | GuardedContextImpl | 278 - org.opendaylight.openflowplugin.impl - 0.5.0.SNAPSHOT | Terminating RpcContextImpl<span class="error">[RUNNING]</span> service for node openflow:109<br/> 2017-09-14 18:01:13,673 | INFO | lt-dispatcher-24 | GuardedContextImpl | 278 - org.opendaylight.openflowplugin.impl - 0.5.0.SNAPSHOT | Terminating StatisticsContextImpl<span class="error">[RUNNING]</span> service for node openflow:109<br/> 2017-09-14 18:01:13,673 | INFO | lt-dispatcher-24 | StatisticsContextImpl | 278 - org.opendaylight.openflowplugin.impl - 0.5.0.SNAPSHOT | Stopping running statistics gathering for node openflow:109<br/> 2017-09-14 18:01:13,673 | WARN | lt-dispatcher-24 | ContextChainHolderImpl | 278 - org.opendaylight.openflowplugin.impl - 0.5.0.SNAPSHOT | Not able to set MASTER role on device openflow:109, reason: Initial gathering statistics unsuccessful.<br/> 2017-09-14 18:01:13,674 | WARN | lt-dispatcher-24 | ContextChainHolderImpl | 278 - org.opendaylight.openflowplugin.impl - 0.5.0.SNAPSHOT | This mastering is mandatory, destroying context chain and closing connection for device openflow:109.<br/> 2017-09-14 18:01:13,674 | INFO | lt-dispatcher-24 | ReconciliationManagerImpl | 271 - org.opendaylight.openflowplugin.applications.reconciliation-framework - 0.5.0.SNAPSHOT | Stopping reconciliation for node Uri <span class="error">[_value=openflow:109]</span><br/> 2017-09-14 18:01:13,674 | INFO | lt-dispatcher-24 | GuardedContextImpl | 278 - org.opendaylight.openflowplugin.impl - 0.5.0.SNAPSHOT | Terminating RoleContextImpl<span class="error">[STARTING]</span> service for node openflow:109<br/> 2017-09-14 18:01:13,674 | INFO | lt-dispatcher-24 | RoleService | 278 - org.opendaylight.openflowplugin.impl - 0.5.0.SNAPSHOT | submitRoleChange onSuccess for device:Uri <span class="error">[_value=openflow:109]</span>, role:BECOMESLAVE<br/> 2017-09-14 18:01:13,674 | WARN | lt-dispatcher-24 | RoleService | 278 - org.opendaylight.openflowplugin.impl - 0.5.0.SNAPSHOT | RpcError on submitRoleChange for Uri <span class="error">[_value=openflow:109]</span>: RpcError <span class="error">[message=Connection closed., severity=ERROR, errorType=APPLICATION, tag=, applicationTag=null, info=null, cause=null]</span><br/> 2017-09-14 18:01:13,676 | INFO | ult-dispatcher-4 | ContextChainHolderImpl | 278 - org.opendaylight.openflowplugin.impl - 0.5.0.SNAPSHOT | Try to remove device Uri <span class="error">[_value=openflow:109]</span> from operational DS<br/> 2017-09-14 18:01:13,683 | INFO | ult-dispatcher-4 | ContextChainHolderImpl | 278 - org.opendaylight.openflowplugin.impl - 0.5.0.SNAPSHOT | Removing device from operational DS Uri <span class="error">[_value=openflow:109]</span> was successful<br/> 2017-09-14 18:01:13,687 | INFO | entLoopGroup-7-8 | RoleService | 278 - org.opendaylight.openflowplugin.impl - 0.5.0.SNAPSHOT | submitRoleChange called for device:Uri <span class="error">[_value=openflow:109]</span>, role:BECOMEMASTER<br/> 2017-09-14 18:01:13,745 | INFO | lt-dispatcher-24 | ContextChainImpl | 278 - org.opendaylight.openflowplugin.impl - 0.5.0.SNAPSHOT | Started clustering services for node openflow:109<br/> 2017-09-14 18:01:14,569 | INFO | entLoopGroup-7-8 | RoleService | 278 - org.opendaylight.openflowplugin.impl - 0.5.0.SNAPSHOT | submitRoleChange onSuccess for device:Uri <span class="error">[_value=openflow:109]</span>, role:BECOMEMASTER<br/> 2017-09-14 18:03:06,220 | INFO | lt-dispatcher-25 | ReconciliationManagerImpl | 271 - org.opendaylight.openflowplugin.applications.reconciliation-framework - 0.5.0.SNAPSHOT | Stopping reconciliation for node Uri <span class="error">[_value=openflow:109]</span><br/> 2017-09-14 18:03:06,220 | INFO | lt-dispatcher-25 | ContextChainHolderImpl | 278 - org.opendaylight.openflowplugin.impl - 0.5.0.SNAPSHOT | Role SLAVE was granted to device openflow:109<br/> 2017-09-14 18:03:06,220 | INFO | lt-dispatcher-25 | GuardedContextImpl | 278 - org.opendaylight.openflowplugin.impl - 0.5.0.SNAPSHOT | Stopping RoleContextImpl<span class="error">[RUNNING]</span> service for node openflow:109<br/> 2017-09-14 18:03:06,220 | INFO | ofppool-40 | ContextChainImpl | 278 - org.opendaylight.openflowplugin.impl - 0.5.0.SNAPSHOT | Closed clustering services for node openflow:109<br/> 2017-09-14 18:15:29,384 | INFO | entLoopGroup-7-8 | SystemNotificationsListenerImpl | 278 - org.opendaylight.openflowplugin.impl - 0.5.0.SNAPSHOT | ConnectionEvent: Connection closed by device, Device:/10.29.13.226:53478, NodeId:openflow:109<br/> 2017-09-14 18:15:29,384 | INFO | entLoopGroup-7-8 | ContextChainHolderImpl | 278 - org.opendaylight.openflowplugin.impl - 0.5.0.SNAPSHOT | Device openflow:109 disconnected.<br/> 2017-09-14 18:15:29,384 | INFO | entLoopGroup-7-8 | ReconciliationManagerImpl | 271 - org.opendaylight.openflowplugin.applications.reconciliation-framework - 0.5.0.SNAPSHOT | Stopping reconciliation for node Uri <span class="error">[_value=openflow:109]</span></p> <p>Thank you for logs, it helped a lot. The failure case was because SLAVE role future was cancelled because we started sending MASTER role, and we sent slave role not aquired notification to mastership service, so I added check for CancellationException so this will not happend anymore. Updated both patches (master, stable/nitrogen), so can you re-test it Louis?</p> <p>Good news last patchset:</p> <p><a href="https://git.opendaylight.org/gerrit/#/c/63123/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/63123/</a></p> <p>gets scalability as should be:</p> <p><a href="https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-1node-periodic-sw-scalability-daily-only-nitrogen/plot/Switch%20Scalability/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-1node-periodic-sw-scalability-daily-only-nitrogen/plot/Switch%20Scalability/</a></p> <p>Nitrogen patch: <a href="https://git.opendaylight.org/gerrit/#/c/63146" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/63146</a></p> <p>Carbon patch: <a href="https://git.opendaylight.org/gerrit/#/c/63275/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/63275/</a><br/> It is a bit different because there is no RoleContext that centralizes role management in Carbon, what complicates things.</p> <p>From your comment: what do we want to do here? are we not going to cherry-pick nitrogen cluster fixes to carbon?</p> <p>Also the carbon patch does not work:</p> <p><a href="https://logs.opendaylight.org/releng/jenkins092/openflowplugin-csit-1node-periodic-sw-scalability-daily-only-carbon/409/odl1_karaf.log.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/openflowplugin-csit-1node-periodic-sw-scalability-daily-only-carbon/409/odl1_karaf.log.gz</a></p> <p>From the log we see connection ongoing but actually nothing shows in operational for switch openflow:146</p> <p>2017-09-20 20:17:15,317 | INFO | entLoopGroup-7-2 | ContextChainHolderImpl | 193 - org.opendaylight.openflowplugin.impl - 0.4.2.SNAPSHOT | Device openflow:146 connected.<br/> 2017-09-20 20:17:15,317 | INFO | entLoopGroup-7-2 | DeviceManagerImpl | 193 - org.opendaylight.openflowplugin.impl - 0.4.2.SNAPSHOT | ConnectionEvent: Device connected to controller, Device:/10.29.12.83:54256, NodeId:Uri <span class="error">[_value=openflow:146]</span><br/> 2017-09-20 20:17:15,317 | INFO | entLoopGroup-7-2 | ContextChainHolderImpl | 193 - org.opendaylight.openflowplugin.impl - 0.4.2.SNAPSHOT | Started timer for setting SLAVE role on node openflow:146 if no role will be set in 20s.<br/> 2017-09-20 20:17:15,317 | INFO | entLoopGroup-7-2 | ContextChainImpl | 193 - org.opendaylight.openflowplugin.impl - 0.4.2.SNAPSHOT | Registering clustering services for node openflow:146<br/> 2017-09-20 20:17:15,317 | INFO | entLoopGroup-7-2 | ContextChainImpl | 193 - org.opendaylight.openflowplugin.impl - 0.4.2.SNAPSHOT | Registered clustering services for node openflow:146<br/> 2017-09-20 20:17:35,196 | INFO | ult-dispatcher-2 | ContextChainImpl | 193 - org.opendaylight.openflowplugin.impl - 0.4.2.SNAPSHOT | Starting clustering services for node openflow:146<br/> 2017-09-20 20:17:35,196 | INFO | ult-dispatcher-2 | DeviceContextImpl | 193 - org.opendaylight.openflowplugin.impl - 0.4.2.SNAPSHOT | Starting device context cluster services for node openflow:146<br/> 2017-09-20 20:17:35,276 | INFO | pool-31-thread-1 | SalRoleServiceImpl | 193 - org.opendaylight.openflowplugin.impl - 0.4.2.SNAPSHOT | SetRole called with input:SetRoleInput [_controllerRole=BECOMESLAVE, _node=NodeRef [_value=KeyedInstanceIdentifier</p> {targetType=interface org.opendaylight.yang.gen.v1.urn.opendaylight.inventory.rev130819.nodes.Node, path=[org.opendaylight.yang.gen.v1.urn.opendaylight.inventory.rev130819.Nodes, org.opendaylight.yang.gen.v1.urn.opendaylight.inventory.rev130819.nodes.Node[key=NodeKey [_id=Uri [_value=openflow:146]]]]} <p>], augmentation=[]]<br/> 2017-09-20 20:17:35,276 | INFO | pool-31-thread-1 | SalRoleServiceImpl | 193 - org.opendaylight.openflowplugin.impl - 0.4.2.SNAPSHOT | RoleChangeTask called on device:openflow:146 OFPRole:BECOMESLAVE<br/> 2017-09-20 20:17:35,276 | INFO | pool-31-thread-1 | RoleService | 193 - org.opendaylight.openflowplugin.impl - 0.4.2.SNAPSHOT | getGenerationIdFromDevice called for device: openflow:146<br/> 2017-09-20 20:17:35,281 | INFO | entLoopGroup-7-2 | DeviceInitializationUtil | 193 - org.opendaylight.openflowplugin.impl - 0.4.2.SNAPSHOT | IP address of the node openflow:146 is: IpAddress [_ipv4Address=Ipv4Address <span class="error">[_value=10.29.12.83]</span>]<br/> 2017-09-20 20:17:35,281 | INFO | entLoopGroup-7-2 | DeviceInitializationUtil | 193 - org.opendaylight.openflowplugin.impl - 0.4.2.SNAPSHOT | Port number of the node openflow:146 is: 54256<br/> 2017-09-20 20:17:35,316 | INFO | entLoopGroup-7-2 | RoleService | 193 - org.opendaylight.openflowplugin.impl - 0.4.2.SNAPSHOT | submitRoleChange called for device:Uri <span class="error">[_value=openflow:146]</span>, role:BECOMESLAVE<br/> 2017-09-20 20:17:35,316 | INFO | entLoopGroup-7-2 | OF13DeviceInitializer | 193 - org.opendaylight.openflowplugin.impl - 0.4.2.SNAPSHOT | Static node openflow:146 info: OFPMPMETERFEATURES collected<br/> 2017-09-20 20:17:35,316 | INFO | entLoopGroup-7-2 | OF13DeviceInitializer | 193 - org.opendaylight.openflowplugin.impl - 0.4.2.SNAPSHOT | Static node openflow:146 info: OFPMPGROUPFEATURES collected<br/> 2017-09-20 20:17:35,316 | INFO | entLoopGroup-7-2 | OF13DeviceInitializer | 193 - org.opendaylight.openflowplugin.impl - 0.4.2.SNAPSHOT | Static node openflow:146 info: OFPMPPORTDESC collected<br/> 2017-09-20 20:17:35,316 | INFO | entLoopGroup-7-2 | OF13DeviceInitializer | 193 - org.opendaylight.openflowplugin.impl - 0.4.2.SNAPSHOT | Static node openflow:146 successfully finished collecting<br/> 2017-09-20 20:17:35,322 | INFO | ult-dispatcher-2 | SalRoleServiceImpl | 193 - org.opendaylight.openflowplugin.impl - 0.4.2.SNAPSHOT | SetRole called with input:SetRoleInput [_controllerRole=BECOMEMASTER, _node=NodeRef [_value=KeyedInstanceIdentifier</p> {targetType=interface org.opendaylight.yang.gen.v1.urn.opendaylight.inventory.rev130819.nodes.Node, path=[org.opendaylight.yang.gen.v1.urn.opendaylight.inventory.rev130819.Nodes, org.opendaylight.yang.gen.v1.urn.opendaylight.inventory.rev130819.nodes.Node[key=NodeKey [_id=Uri [_value=openflow:146]]]]} <p>], augmentation=[]]<br/> 2017-09-20 20:17:35,322 | INFO | ult-dispatcher-2 | SalRoleServiceImpl | 193 - org.opendaylight.openflowplugin.impl - 0.4.2.SNAPSHOT | RoleChangeTask called on device:openflow:146 OFPRole:BECOMEMASTER<br/> 2017-09-20 20:17:35,322 | INFO | ult-dispatcher-2 | RoleService | 193 - org.opendaylight.openflowplugin.impl - 0.4.2.SNAPSHOT | getGenerationIdFromDevice called for device: openflow:146<br/> 2017-09-20 20:17:35,323 | ERROR | ult-dispatcher-2 | SalRoleServiceImpl | 193 - org.opendaylight.openflowplugin.impl - 0.4.2.SNAPSHOT | SetRoleService set Role BECOMESLAVE for Node: KeyedInstanceIdentifier</p> {targetType=interface org.opendaylight.yang.gen.v1.urn.opendaylight.inventory.rev130819.nodes.Node, path=[org.opendaylight.yang.gen.v1.urn.opendaylight.inventory.rev130819.Nodes, org.opendaylight.yang.gen.v1.urn.opendaylight.inventory.rev130819.nodes.Node[key=NodeKey [_id=Uri [_value=openflow:146]]]]} <p> fail . Reason java.util.concurrent.CancellationException: Task was cancelled.<br/> 2017-09-20 20:17:35,323 | INFO | ult-dispatcher-2 | RpcContextImpl | 193 - org.opendaylight.openflowplugin.impl - 0.4.2.SNAPSHOT | Starting rpc context cluster services for node openflow:146<br/> 2017-09-20 20:17:35,332 | INFO | entLoopGroup-7-2 | RoleService | 193 - org.opendaylight.openflowplugin.impl - 0.4.2.SNAPSHOT | submitRoleChange onSuccess for device:Uri <span class="error">[_value=openflow:146]</span>, role:BECOMESLAVE<br/> 2017-09-20 20:17:35,351 | INFO | entLoopGroup-7-2 | RoleService | 193 - org.opendaylight.openflowplugin.impl - 0.4.2.SNAPSHOT | submitRoleChange called for device:Uri <span class="error">[_value=openflow:146]</span>, role:BECOMEMASTER<br/> 2017-09-20 20:17:35,374 | INFO | entLoopGroup-7-2 | RoleService | 193 - org.opendaylight.openflowplugin.impl - 0.4.2.SNAPSHOT | submitRoleChange onSuccess for device:Uri <span class="error">[_value=openflow:146]</span>, role:BECOMEMASTER<br/> 2017-09-20 20:17:35,387 | INFO | ult-dispatcher-2 | StatisticsContextImpl | 193 - org.opendaylight.openflowplugin.impl - 0.4.2.SNAPSHOT | Starting statistics context cluster services for node openflow:146<br/> 2017-09-20 20:17:35,387 | INFO | ult-dispatcher-2 | ContextChainImpl | 193 - org.opendaylight.openflowplugin.impl - 0.4.2.SNAPSHOT | Started clustering services for node openflow:146<br/> 2017-09-20 20:17:35,487 | INFO | entLoopGroup-7-2 | StatisticsManagerImpl | 193 - org.opendaylight.openflowplugin.impl - 0.4.2.SNAPSHOT | Scheduling statistics poll for device: Uri <span class="error">[_value=openflow:146]</span></p> <p>Hmm that is weird, logs looks fine. I noticed some weird thing (that slave task did not got fully cancelled for some reason) so I uploaded new patch.</p> <p>Same issue, still mess up roles in carbon:</p> <p><a href="https://logs.opendaylight.org/releng/jenkins092/openflowplugin-csit-1node-periodic-sw-scalability-daily-only-carbon/411/odl1_karaf.log.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/openflowplugin-csit-1node-periodic-sw-scalability-daily-only-carbon/411/odl1_karaf.log.gz</a></p> <p>Master and Slave at the same time and scale test fails on this switch:</p> <p>2017-09-21 20:59:10,486 | INFO | ult-dispatcher-4 | ContextChainImpl | 193 - org.opendaylight.openflowplugin.impl - 0.4.2.SNAPSHOT | Starting clustering services for node openflow:73<br/> 2017-09-21 20:59:10,486 | INFO | ult-dispatcher-4 | DeviceContextImpl | 193 - org.opendaylight.openflowplugin.impl - 0.4.2.SNAPSHOT | Starting device context cluster services for node openflow:73<br/> 2017-09-21 20:59:10,634 | INFO | pool-28-thread-1 | SalRoleServiceImpl | 193 - org.opendaylight.openflowplugin.impl - 0.4.2.SNAPSHOT | SetRole called with input:SetRoleInput [_controllerRole=BECOMESLAVE, _node=NodeRef [_value=KeyedInstanceIdentifier</p> {targetType=interface org.opendaylight.yang.gen.v1.urn.opendaylight.inventory.rev130819.nodes.Node, path=[org.opendaylight.yang.gen.v1.urn.opendaylight.inventory.rev130819.Nodes, org.opendaylight.yang.gen.v1.urn.opendaylight.inventory.rev130819.nodes.Node[key=NodeKey [_id=Uri [_value=openflow:73]]]]} <p>], augmentation=[]]<br/> 2017-09-21 20:59:10,634 | INFO | pool-28-thread-1 | SalRoleServiceImpl | 193 - org.opendaylight.openflowplugin.impl - 0.4.2.SNAPSHOT | RoleChangeTask called on device:openflow:73 OFPRole:BECOMESLAVE<br/> 2017-09-21 20:59:10,634 | INFO | pool-28-thread-1 | RoleService | 193 - org.opendaylight.openflowplugin.impl - 0.4.2.SNAPSHOT | getGenerationIdFromDevice called for device: openflow:73<br/> 2017-09-21 20:59:10,662 | INFO | entLoopGroup-7-3 | DeviceInitializationUtil | 193 - org.opendaylight.openflowplugin.impl - 0.4.2.SNAPSHOT | IP address of the node openflow:73 is: IpAddress [_ipv4Address=Ipv4Address <span class="error">[_value=10.29.15.165]</span>]<br/> 2017-09-21 20:59:10,662 | INFO | entLoopGroup-7-3 | DeviceInitializationUtil | 193 - org.opendaylight.openflowplugin.impl - 0.4.2.SNAPSHOT | Port number of the node openflow:73 is: 36476<br/> 2017-09-21 20:59:10,726 | INFO | entLoopGroup-7-3 | RoleService | 193 - org.opendaylight.openflowplugin.impl - 0.4.2.SNAPSHOT | submitRoleChange called for device:Uri <span class="error">[_value=openflow:73]</span>, role:BECOMESLAVE<br/> 2017-09-21 20:59:10,726 | INFO | entLoopGroup-7-3 | OF13DeviceInitializer | 193 - org.opendaylight.openflowplugin.impl - 0.4.2.SNAPSHOT | Static node openflow:73 info: OFPMPMETERFEATURES collected<br/> 2017-09-21 20:59:10,726 | INFO | entLoopGroup-7-3 | OF13DeviceInitializer | 193 - org.opendaylight.openflowplugin.impl - 0.4.2.SNAPSHOT | Static node openflow:73 info: OFPMPGROUPFEATURES collected<br/> 2017-09-21 20:59:10,726 | INFO | entLoopGroup-7-3 | OF13DeviceInitializer | 193 - org.opendaylight.openflowplugin.impl - 0.4.2.SNAPSHOT | Static node openflow:73 info: OFPMPPORTDESC collected<br/> 2017-09-21 20:59:10,726 | INFO | entLoopGroup-7-3 | OF13DeviceInitializer | 193 - org.opendaylight.openflowplugin.impl - 0.4.2.SNAPSHOT | Static node openflow:73 successfully finished collecting<br/> 2017-09-21 20:59:10,729 | INFO | ult-dispatcher-4 | SalRoleServiceImpl | 193 - org.opendaylight.openflowplugin.impl - 0.4.2.SNAPSHOT | SetRole called with input:SetRoleInput [_controllerRole=BECOMEMASTER, _node=NodeRef [_value=KeyedInstanceIdentifier</p> {targetType=interface org.opendaylight.yang.gen.v1.urn.opendaylight.inventory.rev130819.nodes.Node, path=[org.opendaylight.yang.gen.v1.urn.opendaylight.inventory.rev130819.Nodes, org.opendaylight.yang.gen.v1.urn.opendaylight.inventory.rev130819.nodes.Node[key=NodeKey [_id=Uri [_value=openflow:73]]]]} <p>], augmentation=[]]<br/> 2017-09-21 20:59:10,729 | INFO | ult-dispatcher-4 | SalRoleServiceImpl | 193 - org.opendaylight.openflowplugin.impl - 0.4.2.SNAPSHOT | RoleChangeTask called on device:openflow:73 OFPRole:BECOMEMASTER<br/> 2017-09-21 20:59:10,729 | INFO | ult-dispatcher-4 | RoleService | 193 - org.opendaylight.openflowplugin.impl - 0.4.2.SNAPSHOT | getGenerationIdFromDevice called for device: openflow:73<br/> 2017-09-21 20:59:10,729 | INFO | ult-dispatcher-4 | RpcContextImpl | 193 - org.opendaylight.openflowplugin.impl - 0.4.2.SNAPSHOT | Starting rpc context cluster services for node openflow:73<br/> 2017-09-21 20:59:10,735 | INFO | ult-dispatcher-4 | StatisticsContextImpl | 193 - org.opendaylight.openflowplugin.impl - 0.4.2.SNAPSHOT | Starting statistics context cluster services for node openflow:73<br/> 2017-09-21 20:59:10,736 | INFO | ult-dispatcher-4 | ContextChainImpl | 193 - org.opendaylight.openflowplugin.impl - 0.4.2.SNAPSHOT | Started clustering services for node openflow:73<br/> 2017-09-21 20:59:10,853 | INFO | entLoopGroup-7-3 | RoleService | 193 - org.opendaylight.openflowplugin.impl - 0.4.2.SNAPSHOT | submitRoleChange onSuccess for device:Uri <span class="error">[_value=openflow:73]</span>, role:BECOMESLAVE<br/> 2017-09-21 20:59:10,853 | INFO | entLoopGroup-7-3 | RoleService | 193 - org.opendaylight.openflowplugin.impl - 0.4.2.SNAPSHOT | submitRoleChange called for device:Uri <span class="error">[_value=openflow:73]</span>, role:BECOMEMASTER<br/> 2017-09-21 20:59:10,861 | INFO | entLoopGroup-7-3 | RoleService | 193 - org.opendaylight.openflowplugin.impl - 0.4.2.SNAPSHOT | submitRoleChange onSuccess for device:Uri <span class="error">[_value=openflow:73]</span>, role:BECOMEMASTER<br/> 2017-09-21 20:59:10,926 | INFO | entLoopGroup-7-3 | StatisticsManagerImpl | 193 - org.opendaylight.openflowplugin.impl - 0.4.2.SNAPSHOT | Scheduling statistics poll for device: Uri <span class="error">[_value=openflow:73]</span></p> <p>Change title to reflect only carbon has the issue now.</p> <p>Alright, I tried to reproduce what was changed in role context in nitrogen and master, can you try this issue with this patch: <a href="https://git.opendaylight.org/gerrit/#/c/63536/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/63536/</a> luis?</p> <p>OK, the last patch seems to work in carbon but now we have new issue preventing switch scale test to work properly:</p> <p><a href="https://bugs.opendaylight.org/show_bug.cgi?id=9217" class="external-link" target="_blank" rel="nofollow noopener">https://bugs.opendaylight.org/show_bug.cgi?id=9217</a></p> <p>Alright the patch on carbon got merged, so should we wait until the new bug is closed before we close this one?</p> <p>No thats fine, I already tested it.</p> Development External issue ID 9089 External issue URL Rank 0|i0348v: