[OPNFLWPLUG-666] Li Plugin only: switch unregister failure in OF HA test Created: 01/Apr/16  Updated: 27/Sep/21  Resolved: 25/Aug/16

Status: Resolved
Project: OpenFlowPlugin
Component/s: General
Affects Version/s: None
Fix Version/s: None

Type: Bug
Reporter: Luis Gomez Assignee: Unassigned
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Operating System: All
Platform: All

Issue Links:
Blocks
is blocked by OPNFLWPLUG-659 Milestone: integrate with Singleton a... Resolved
External issue ID: 5641

 Description   

We have sporadic OF cluster failure (both He and Li plugins) after restarting single node few times and disconnecting mininet.

The sympthom is a switch shows in operational inventory while it is not anymore connected (it does not show either in entity owner).

A tail of the karaf log on the device MASTER node shows continuous WARN messages (see last lines) after openflow:1 switch has been disconnected:

2016-04-01 00:30:07,099 | INFO | entLoopGroup-5-4 | OfEntityManager | 177 - org.opendaylight.openflowplugin - 0.3.0.SNAPSHOT | unregisterEntityOwnershipRequest: Unregister controller entity ownership request for Uri [_value=openflow:1]
2016-04-01 00:30:07,111 | WARN | on-dispatcher-72 | ConnectorInventoryTranslatorImpl | 187 - org.opendaylight.openflowplugin.applications.forwardingrules-manager - 0.3.0.SNAPSHOT | Node Connector removed
2016-04-01 00:30:07,116 | INFO | pool-28-thread-1 | StatisticsManagerImpl | 182 - org.opendaylight.openflowplugin.applications.statistics-manager - 0.3.0.SNAPSHOT | Node:KeyedInstanceIdentifier

{targetType=interface org.opendaylight.yang.gen.v1.urn.opendaylight.inventory.rev130819.nodes.Node, path=[org.opendaylight.yang.gen.v1.urn.opendaylight.inventory.rev130819.Nodes, org.opendaylight.yang.gen.v1.urn.opendaylight.inventory.rev130819.nodes.Node[key=NodeKey [_id=Uri [_value=openflow:1]]]]}

successfully removed by StatisticsManager
2016-04-01 00:30:07,145 | WARN | on-dispatcher-68 | ConnectorInventoryTranslatorImpl | 187 - org.opendaylight.openflowplugin.applications.forwardingrules-manager - 0.3.0.SNAPSHOT | Node Connector updated
2016-04-01 00:30:07,148 | WARN | on-dispatcher-69 | ConnectorInventoryTranslatorImpl | 187 - org.opendaylight.openflowplugin.applications.forwardingrules-manager - 0.3.0.SNAPSHOT | Node Connector removed
2016-04-01 00:30:07,152 | INFO | entLoopGroup-5-3 | OfEntityManager | 177 - org.opendaylight.openflowplugin - 0.3.0.SNAPSHOT | unregisterEntityOwnershipRequest: Unregister controller entity ownership request for Uri [_value=openflow:2]
2016-04-01 00:30:07,158 | INFO | pool-28-thread-1 | StatisticsManagerImpl | 182 - org.opendaylight.openflowplugin.applications.statistics-manager - 0.3.0.SNAPSHOT | Node:KeyedInstanceIdentifier

{targetType=interface org.opendaylight.yang.gen.v1.urn.opendaylight.inventory.rev130819.nodes.Node, path=[org.opendaylight.yang.gen.v1.urn.opendaylight.inventory.rev130819.Nodes, org.opendaylight.yang.gen.v1.urn.opendaylight.inventory.rev130819.nodes.Node[key=NodeKey [_id=Uri [_value=openflow:2]]]]}

successfully removed by StatisticsManager
2016-04-01 00:30:07,164 | WARN | on-dispatcher-68 | ConnectorInventoryTranslatorImpl | 187 - org.opendaylight.openflowplugin.applications.forwardingrules-manager - 0.3.0.SNAPSHOT | Node Connector removed
2016-04-01 00:30:07,164 | WARN | on-dispatcher-68 | ConnectorInventoryTranslatorImpl | 187 - org.opendaylight.openflowplugin.applications.forwardingrules-manager - 0.3.0.SNAPSHOT | Node Connector removed
2016-04-01 00:30:07,164 | WARN | on-dispatcher-68 | ConnectorInventoryTranslatorImpl | 187 - org.opendaylight.openflowplugin.applications.forwardingrules-manager - 0.3.0.SNAPSHOT | Node Connector removed
2016-04-01 00:30:07,227 | INFO | entLoopGroup-5-4 | OfEntityManager | 177 - org.opendaylight.openflowplugin - 0.3.0.SNAPSHOT | unregisterEntityOwnershipRequest: Unregister controller entity ownership request for Uri [_value=openflow:3]
2016-04-01 00:30:07,229 | INFO | pool-28-thread-1 | StatisticsManagerImpl | 182 - org.opendaylight.openflowplugin.applications.statistics-manager - 0.3.0.SNAPSHOT | Node:KeyedInstanceIdentifier

{targetType=interface org.opendaylight.yang.gen.v1.urn.opendaylight.inventory.rev130819.nodes.Node, path=[org.opendaylight.yang.gen.v1.urn.opendaylight.inventory.rev130819.Nodes, org.opendaylight.yang.gen.v1.urn.opendaylight.inventory.rev130819.nodes.Node[key=NodeKey [_id=Uri [_value=openflow:3]]]]}

successfully removed by StatisticsManager
2016-04-01 00:30:07,234 | WARN | on-dispatcher-69 | ConnectorInventoryTranslatorImpl | 187 - org.opendaylight.openflowplugin.applications.forwardingrules-manager - 0.3.0.SNAPSHOT | Node Connector removed
2016-04-01 00:30:07,234 | WARN | on-dispatcher-69 | ConnectorInventoryTranslatorImpl | 187 - org.opendaylight.openflowplugin.applications.forwardingrules-manager - 0.3.0.SNAPSHOT | Node Connector removed
2016-04-01 00:30:07,234 | WARN | on-dispatcher-69 | ConnectorInventoryTranslatorImpl | 187 - org.opendaylight.openflowplugin.applications.forwardingrules-manager - 0.3.0.SNAPSHOT | Node Connector removed
2016-04-01 00:30:16,299 | WARN | ult-dispatcher-2 | RpcRegistry | 165 - org.opendaylight.controller.sal-remoterpc-connector - 1.4.0.SNAPSHOT | Timed out finding routers for RouteIdentifierImpl{context=null, type=(urn:opendaylight:packet:service?revision=2013-07-09)transmit-packet, route=/(urn:opendaylight:inventory?revision=2013-08-19)nodes/node/node[

{(urn:opendaylight:inventory?revision=2013-08-19)id=openflow:1}

]}
2016-04-01 00:30:16,300 | WARN | ult-dispatcher-2 | RpcRegistry | 165 - org.opendaylight.controller.sal-remoterpc-connector - 1.4.0.SNAPSHOT | Timed out finding routers for RouteIdentifierImpl{context=null, type=(urn:opendaylight:packet:service?revision=2013-07-09)transmit-packet, route=/(urn:opendaylight:inventory?revision=2013-08-19)nodes/node/node[

{(urn:opendaylight:inventory?revision=2013-08-19)id=openflow:1}

]}
2016-04-01 00:30:21,316 | WARN | ult-dispatcher-4 | RpcRegistry | 165 - org.opendaylight.controller.sal-remoterpc-connector - 1.4.0.SNAPSHOT | Timed out finding routers for RouteIdentifierImpl{context=null, type=(urn:opendaylight:packet:service?revision=2013-07-09)transmit-packet, route=/(urn:opendaylight:inventory?revision=2013-08-19)nodes/node/node[

{(urn:opendaylight:inventory?revision=2013-08-19)id=openflow:1}

]}
2016-04-01 00:30:21,316 | WARN | ult-dispatcher-4 | RpcRegistry | 165 - org.opendaylight.controller.sal-remoterpc-connector - 1.4.0.SNAPSHOT | Timed out finding routers for RouteIdentifierImpl{context=null, type=(urn:opendaylight:packet:service?revision=2013-07-09)transmit-packet, route=/(urn:opendaylight:inventory?revision=2013-08-19)nodes/node/node[

{(urn:opendaylight:inventory?revision=2013-08-19)id=openflow:1}

]}

 Comments   
Comment by Shuva Jyoti Kar [ 12/Jun/16 ]

Hey Luis,

A couple of questions:

1. Are you running mininet in OF-HA mode ?
2. Is the switch getting removed and re-added to the same controller node or a different one ?

thanks
Shuva

Comment by Shuva Jyoti Kar [ 12/Jun/16 ]

Oops,sorry i missed the bug topic. I think the answer to the first question is yes.
So do you keep disconnecting the master connection/connections across all the nodes ?

Comment by Luis Gomez [ 14/Jun/16 ]

This is in the OF HA test so the switch is connected and then disconnected from the 3 nodes.

Comment by Luis Gomez [ 08/Jul/16 ]

This is only happening with Li plugin.

To reproduce:

  • Start cluster 3 nodes.
  • Connect switch to 3 nodes.
  • Find out switch owner.
  • Kill the switch owner.
  • Start old switch owner.
  • Disonnect switch.
  • Check inventory information.

BR/Luis

Comment by Tomas Slusny [ 04/Aug/16 ]

I was able to reproduce this issue on master. Inventory was not properly cleared and switch was still there. Then I tested this with Jozef's patches, and switch was properly removed from it (one empty node was still there, but I think it is not related to this bug). So I believe, that https://bugs.opendaylight.org/show_bug.cgi?id=5596 will resolve this.

Comment by Tomas Slusny [ 25/Aug/16 ]

Tested this again on latest master with Jozef's patches merged. Everything is working perfcetly fine, after reproducing steps by Luis, inventory was perfectly clean.

Comment by Luis Gomez [ 25/Aug/16 ]

Agree, I also tested this yesterday and worked.

Generated at Wed Feb 07 20:33:04 UTC 2024 using Jira 8.20.10#820010-sha1:ace47f9899e9ee25d7157d59aa17ab06aee30d3d.