[NETVIRT-1065] Patch ports are not created between br-int and br-physnet1 Created: 16/Jan/18  Updated: 22/Feb/18  Resolved: 22/Feb/18

Status: Resolved
Project: netvirt
Component/s: General
Affects Version/s: Nitrogen, Carbon, Oxygen
Fix Version/s: Oxygen

Type: Bug Priority: Highest
Reporter: Sam Hague Assignee: Josh Hershberg
Resolution: Done Votes: 0
Labels: csit:failures
Remaining Estimate: 0 minutes
Time Spent: 1 day
Original Estimate: Not Specified


 Description   

https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-openstack-ocata-upstream-stateful-nitrogen/771/log_01_l2.html.gz
 
See two issues with patch ports between br-int and br-physnet1:
 
1. They are never created
2. They take over 90s to be created
 [1] is a job where the patch ports fail to add on compute_2. What I also notice is from the time br-physnet1 is created until patch ports are created on control_1 is 30s, then for compute_1 it is 100s. And then compute_2 never tries to create the patch ports.
 
Recall the flow in csit:
0. Start stacking
1. Nodes connect to ODL
2. netvirt adds br-int and adds br-physnet1 port to br-int
3. Stacking completes
4. Delete br-physnet1 from br-int
5. Add br-physnet1 bridge
6. netvirt adds patch ports
 
That all seems to work for the control and compute_1 but fails on the patch ports for compute_2. And as noted above, it took 37s to add them on the control node and 100s to add them on the control_1. Below for each node is the log when the br-physnet1 is seen and the next log is adding the patch ports.
 
Can anyone see anything strange in the logs that would explain the issue? This happens randomly on either node or sometimes both nodes.
 
I see in [2] which is the ovsdb log, that during this time when the ports should be updated that something is trying to create br-int, which you can see in the "record 24" and "record 25" logs below. It says invoked by user/python so that leads me to believe it is openstack code and not ODL. That seems to trigger the ovsdbnode update in ODL for compute 1, but doesn't help on compute 2. on compute 2 it happened just as tap ports were being added so maybe that casues problems.
 
So it seems the patch to make sure the patch ports are created is missing something since it is not triggering the creation.
 
 

 control_1:    2018-01-14 23:27:10,854 | INFO  | eChangeHandler-0 | OvsdbNodeListener                | 242 - org.opendaylight.genius.itm-impl - 0.3.2.SNAPSHOT | OvsdbBridgeAugmentation ADD: DPID for bridge br-physnet1 is NULL. 2018-01-14 23:27:57,483 | INFO  | eChangeHandler-0 | ElanBridgeManager                | 314 - org.opendaylight.netvirt.elanmanager-impl - 0.5.2.SNAPSHOT | addPatchPort: node: ovsdb://uuid/72adceaf-e656-432f-8d49-6eab88d40d06/bridge/br-int, bridge: br-int, portname: br-physnet1-pa peer: br-physnet1-int-patch status: success    compute_1:
 2018-01-14 23:27:12,127 | INFO  | eChangeHandler-0 | OvsdbNodeListener                | 242 - org.opendaylight.genius.itm-impl - 0.3.2.SNAPSHOT | OvsdbBridgeAugmentation ADD: DPID for bridge br-physnet1 is NULL.
  record 24: 2018-01-14 23:28:56.501 "ovs-vsctl (invoked by /bin/python): ovs-vsctl -- --may-exist add-br br-int -- set Bridge br-int datapath_type=system"    2018-01-14 23:28:56,524 | INFO  | eChangeHandler-0 | ElanBridgeManager                | 314 - org.opendaylight.netvirt.elanmanager-impl - 0.5.2.SNAPSHOT | addPatchPort: node: ovsdb://uuid/c2e6d059-7ae8-4c4c-92fb-c40b656f5bdb/bridge/br-int, bridge: br-int, portname: br-physnet1-pa peer: br-physnet1-int-patch status: success
   compute_2: 2018-01-14 23:27:13,326 | INFO  | eChangeHandler-0 | OvsdbNodeListener                | 242 - org.opendaylight.genius.itm-impl - 0.3.2.SNAPSHOT | OvsdbBridgeAugmentation ADD: DPID for bridge br-physnet1 is NULL.
  record 25: 2018-01-14 23:29:05.581 "ovs-vsctl (invoked by /bin/python): ovs-vsctl -- --may-exist add-br br-int -- set Bridge br-int datapath_type=system"  

 
[1] https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-openstack-ocata-upstream-stateful-nitrogen/771/log_01_l2.html.gz
 
[2] https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-openstack-ocata-upstream-stateful-nitrogen/771/compute_2/extra_debug.log.gz

 Comments   
Comment by Sam Hague [ 16/Jan/18 ]

62347 introduced a change where bridge updates other than br-int were dropped. Before that patch went in, when br-physnet1 was created an update would come in and the patch port creation path would execute. After that patch, this path was dropped. Only when an update comes in much later for br-int do we add the patch-ports. Another issue is that in some cases we don't get any notifications so the patch port path never executes.

[4] is a job that is before the patches:

 

2017-09-11 12:35:38,990 | INFO | eChangeHandler-0 | OvsdbNodeListener | 240 - org.opendaylight.genius.itm-impl - 0.4.0.SNAPSHOT | OvsdbBridgeAugmentation ADD: DPID for bridge br-physnet1 is NULL. 2017-09-11 12:35:38,992 | WARN | nPool-2-worker-0 | sInterfaceTopologyStateAddHelper | 237 - org.opendaylight.genius.interfacemanager-impl - 0.4.0.SNAPSHOT | DataPathId found as null for Bridge Augmentation: OvsdbBridgeAugmentation{getBridgeName=OvsdbBridgeName [_value=br-physnet1], getBridgeOtherConfigs=[BridgeOtherConfigs{getBridgeOtherConfigKey=hwaddr, getBridgeOtherConfigValue=f6:00:00:ff:01:01, augmentations={}}, BridgeOtherConfigs{getBridgeOtherConfigKey=disable-in-band, getBridgeOtherConfigValue=true, augmentations={}}], getBridgeUuid=Uuid [_value=0abface2-b5e0-48e7-bf99-14fb45b3f18c], getDatapathType=class org.opendaylight.yang.gen.v1.urn.opendaylight.params.xml.ns.yang.ovsdb.rev150105.DatapathTypeSystem, getManagedBy=OvsdbNodeRef [_value=KeyedInstanceIdentifier{targetType=interface org.opendaylight.yang.gen.v1.urn.tbd.params.xml.ns.yang.network.topology.rev131021.network.topology.topology.Node, path=[org.opendaylight.yang.gen.v1.urn.tbd.params.xml.ns.yang.network.topology.rev131021.NetworkTopology, org.opendaylight.yang.gen.v1.urn.tbd.params.xml.ns.yang.network.topology.rev131021.network.topology.Topology[key=TopologyKey [_topologyId=Uri [_value=ovsdb:1]]], org.opendaylight.yang.gen.v1.urn.tbd.params.xml.ns.yang.network.topology.rev131021.network.topology.topology.Node[key=NodeKey [_nodeId=Uri [_value=ovsdb://uuid/8054e9a4-3550-4c00-9084-3d4aeb884ec8]]]]}], isStpEnable=false}... returning... 2017-09-11 12:35:38,996 | INFO | eChangeHandler-0 | ElanBridgeManager | 313 - org.opendaylight.netvirt.elanmanager-impl - 0.6.0.SNAPSHOT | addPatchPort: node: ovsdb://uuid/8054e9a4-3550-4c00-9084-3d4aeb884ec8/bridge/br-int, bridge: br-int, portname: br-physnet1-pa peer: br-physnet1-int-patch status: success
2017-09-11 12:35:38,996 | INFO | eChangeHandler-0 | ElanBridgeManager | 313 - org.opendaylight.netvirt.elanmanager-impl - 0.6.0.SNAPSHOT | addPatchPort: node: ovsdb://uuid/8054e9a4-3550-4c00-9084-3d4aeb884ec8/bridge/br-int, bridge: br-int, portname: br-physnet1-pa peer: br-physnet1-int-patch status: success 

[5]. is after the patch and now the br-physnet1 update is dropped so patch ports are not created:

 

2017-09-18 08:13:04,304 | INFO | eChangeHandler-0 | OvsdbNodeListener | 240 - org.opendaylight.genius.itm-impl - 0.4.0.SNAPSHOT | OvsdbBridgeAugmentation ADD: DPID for bridge br-physnet1 is NULL. 2017-09-18 08:13:04,304 | WARN | nPool-2-worker-3 | sInterfaceTopologyStateAddHelper | 237 - org.opendaylight.genius.interfacemanager-impl - 0.4.0.SNAPSHOT | DataPathId found as null for Bridge Augmentation: OvsdbBridgeAugmentation{getBridgeName=OvsdbBridgeName [_value=br-physnet1], getBridgeOtherConfigs=[BridgeOtherConfigs{getBridgeOtherConfigKey=hwaddr, getBridgeOtherConfigValue=f6:00:00:ff:01:03, augmentations={}}, BridgeOtherConfigs{getBridgeOtherConfigKey=disable-in-band, getBridgeOtherConfigValue=true, augmentations={}}], getBridgeUuid=Uuid [_value=a881918a-f9e5-497d-97c2-370f0f64c823], getDatapathType=class org.opendaylight.yang.gen.v1.urn.opendaylight.params.xml.ns.yang.ovsdb.rev150105.DatapathTypeSystem, getManagedBy=OvsdbNodeRef [_value=KeyedInstanceIdentifier{targetType=interface org.opendaylight.yang.gen.v1.urn.tbd.params.xml.ns.yang.network.topology.rev131021.network.topology.topology.Node, path=[org.opendaylight.yang.gen.v1.urn.tbd.params.xml.ns.yang.network.topology.rev131021.NetworkTopology, org.opendaylight.yang.gen.v1.urn.tbd.params.xml.ns.yang.network.topology.rev131021.network.topology.Topology[key=TopologyKey [_topologyId=Uri [_value=ovsdb:1]]], org.opendaylight.yang.gen.v1.urn.tbd.params.xml.ns.yang.network.topology.rev131021.network.topology.topology.Node[key=NodeKey [_nodeId=Uri [_value=ovsdb://uuid/6ca90e85-3571-490a-ba9a-5b64460c7f14]]]]}], isStpEnable=false}... returning...
...
2017-09-18 08:13:31,477 | INFO | eChangeHandler-0 | ElanBridgeManager | 316 - org.opendaylight.netvirt.elanmanager-impl - 0.6.0.SNAPSHOT | addPatchPort: node: ovsdb://uuid/45769152-dc03-4487-b029-9f8bf651981a/bridge/br-int, bridge: br-int, portname: br-physnet1-pa peer: br-physnet1-int-patch status: success

 

 

[3] https://git.opendaylight.org/gerrit/#/c/62347/

[4] https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-ocata-gate-stateful-oxygen/1/odl1_karaf.log.gz

[5] https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-ocata-gate-stateful-oxygen/28/odl1_karaf.log.gz

Comment by Sam Hague [ 16/Jan/18 ]

Some options:

  1. enforce that bridges be name br-xxx so that netvirt can distinguish within provider_mappings if the user expects a bridge or a port on br-int. provider_mappings=physnet1:br-physnet1. In OS land, bridge_mappings is used and it is clear that it is only for bridges. They do not have the concept of adding a port directly to br-int.
  2. Add path in netvirt to process bridges in the provider_mappings. bridge update comes in, check if in provider_mappings and then build patch ports.
  3. Drop the port option, but still need to add 2.
  4. Modify CSIT to create br-physnet1 before stacking so that when br-int is created it will add the patch ports. This would still leave the issue of not creating patch ports if the user adds the provider bridges later.
  5. Add explicit option for provider_port_mappings. Or switch to using host-config.

 

Generated at Wed Feb 07 20:23:09 UTC 2024 using Jira 8.20.10#820010-sha1:ace47f9899e9ee25d7157d59aa17ab06aee30d3d.