[OVSDB-419] L2GW LogicalSwitch tunnel is not created Created: 14/Jun/17  Updated: 30/Oct/17  Resolved: 04/Jul/17

Status: Resolved
Project: ovsdb
Component/s: Southbound.hw_vtep
Affects Version/s: unspecified
Fix Version/s: None

Type: Bug
Reporter: Ran Xiao Assignee: Vishal Thapar
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Operating System: All
Platform: All

Issue Links:
Duplicate
duplicates OVSDB-418 port vlan bindings reconciliation is ... Resolved
External issue ID: 8690

 Description   

L2GW LogicalSwitch tunnel is not created after executed l2gateway connection creation command.
Command executed: neutron l2-gateway-connection-create
Performed 100 times and only succeeded once.

Environment details:
OpenStack Version：stable/ocata
ODL Version：Carbon-FR + patch
patch: https://git.opendaylight.org/gerrit/#/c/56773/
https://git.opendaylight.org/gerrit/#/c/56710/
HWVTEP: Open vSwitch 2.6.1 HWVTEP Emulator
: HA Cluster

The following logs are found in Karaf.log
2017-06-08 14:45:45,804 | ERROR | -dispatcher-7716 | TransactInvokerImpl | 292 - org.opendaylight.ovsdb.hwvtepsouthbound-impl - 1.4.0.Carbon | HWVTEP transaction operation failed referential integrity violation Table Ucast_Macs_Remote column locator row 3ba6a92d-52a0-46e3-ae41-dedb95da8e74 references nonexistent row 00000000-0000-0000-0000-000000000000 in table Physical_Locator.
2017-06-08 14:45:45,804 | ERROR | -dispatcher-7716 | TransactInvokerImpl | 292 - org.opendaylight.ovsdb.hwvtepsouthbound-impl - 1.4.0.Carbon | Failed transaction update [vlan_bindings :

{2222=f7617bb2-98ca-4ba0-9c61-94df217e29c6}

, ]insert [encapsulation_type : vxlan_over_ipv4 , dst_ip : 10.0.0.10 , ] insert [locators : [PhysicalLocator_Random_8b95ac7a75414bfeaed47dc246aeadda] , ] insert [locator_set : PhysicalLocatorSet_Random_82148b021d94464bb06049e90e1b4e67 , logical_switch : f7617bb2-98ca-4ba0-9c61-94df217e29c6 , MAC : unknown-dst , ] insert [logical_switch : f7617bb2-98ca-4ba0-9c61-94df217e29c6 , MAC : fa:16:3e:ea:8b:fe , ]

Cause of failure:
'locator' is not specified for 'insert [logical_switch : f7617bb2-98ca-4ba0-9c61-94df217e29c6 , MAC : fa:16:3e:ea:8b:fe , ]'.
I think this caused the failure of insert operation of Ucast_Macs_Remote table.

When tunnel was created successfully, there were two transactions in ovsdb.
And 'locator' was specified and insert operation of Ucast_Macs_Remote table was succeed at that time.
1. the 1st transaction
update [vlan_bindings :

{2222=53968c22-9106-4f5e-9d3b-84713fe3bda7}

, ]
insert [encapsulation_type : vxlan_over_ipv4 , dst_ip : 10.0.0.10 , ]
insert [locators : [PhysicalLocator_Random_5aa0854faa6d42169564a904de0d45d9] , ]
insert [locator_set : PhysicalLocatorSet_Random_4812e00980db4d68909451b1fe94db8a , logical_switch : 53968c22-9106-4f5e-9d3b-84713fe3bda7 , MAC : unknown-dst , ]
2. the 2nd transaction
insert [locator : 00d1ef47-faf1-4d63-8f52-ad4121d9a235 , logical_switch : 53968c22-9106-4f5e-9d3b-84713fe3bda7 , MAC : fa:16:3e:58:ba:81 , ]

 Comments   
Comment by Ran Xiao [ 16/Jun/17 ]

UPDATES:

The following error occurred when we adding HWVTEP Node.
It seems to be the same cause.

Errors:
2017-06-14 14:14:35,554 | ERROR | on-dispatcher-66 | TransactInvokerImpl | 292 - org.opendaylight.ovsdb.hwvtepsouthbound-impl - 1.4.1.SNAPSHOT | HWVTEP transaction operation failed referential integrity violation Table Physical_Port column vlan_bindings row d400dc78-d0b2-4a45-beba-326b18eb4d59 references nonexistent row cc7cbc81-eb4d-4f2b-ba04-9dfa6afe849d in table Logical_Switch.
2017-06-14 14:14:35,555 | ERROR | on-dispatcher-66 | TransactInvokerImpl | 292 - org.opendaylight.ovsdb.hwvtepsouthbound-impl - 1.4.1.SNAPSHOT | Failed transaction update [tunnel_ips : [] , ]update [vlan_bindings :

{2222=LogicalSwitch_30ae2d8c_5361_45be_b7a8_a93abf722e8f} , ]
2017-06-14 14:14:36,047 | ERROR | on-dispatcher-66 | TransactInvokerImpl | 292 - org.opendaylight.ovsdb.hwvtepsouthbound-impl - 1.4.1.SNAPSHOT | HWVTEP transaction operation failed referential integrity violation Table Ucast_Macs_Remote column locator row 17ddb12a-f4de-487d-908f-1717790100f1 references nonexistent row 00000000-0000-0000-0000-000000000000 in table Physical_Locator.
2017-06-14 14:14:36,047 | ERROR | on-dispatcher-66 | TransactInvokerImpl | 292 - org.opendaylight.ovsdb.hwvtepsouthbound-impl - 1.4.1.SNAPSHOT | Failed transaction insert [name : 30ae2d8c-5361-45be-b7a8-a93abf722e8f , tunnel_key : [96] , ] insert [encapsulation_type : vxlan_over_ipv4 , dst_ip : 10.0.0.10 , ] insert [locators : [PhysicalLocator_Random_61098e5741e640c6a118896b0c87e1c9] , ] insert [locator_set : PhysicalLocatorSet_Random_49a5353230ee467c8a8254689b15a162 , logical_switch : LogicalSwitch_30ae2d8c_5361_45be_b7a8_a93abf722e8f , MAC : unknown-dst , ] insert [logical_switch : LogicalSwitch_30ae2d8c_5361_45be_b7a8_a93abf722e8f , MAC : fa:16:3e:fa:1d:10 , ]

Cause of failure:
The operation order of OVSDB is incorrect.

The uuid of LogicalSwitch_30ae2d8c_5361_45be_b7a8_a93abf722e8f will not be assigned if update [vlan_bindings : {2222=LogicalSwitch_30ae2d8c_5361_45be_b7a8_a93abf722e8f}

, ] is not executed after Logical Switch creation with operation of insert [name : 30ae2d8c-5361-45be-b7a8-a93abf722e8f , tunnel_key : [96] , ]

Root cause:
The order of OVSDB operations are not considered

Detailed info of adding HWVTEP Node
Steps executed:
1. In HWVTEP#1
Create GATEWAY and CONNECTION for this HWVTEP.
Confirmed the communication from DHCP namespace to Baremetal is OK.
2. In new node, HWVTEP#2
Start up with the same parameters as HWVTEP#1 using.
And set up HA Cluster.
For 'other_config:ha_id', 'vtep', 'Physical Switch name', we specified the same value for HWVTEP#1 and #2.
'other_config:ha_enabled' is set to 'true'

Commans executed:
---- HWVTEP#1/HWVTEP#2 ----
vtep=10.0.0.50
BRIDGE=ocata-l2gw1
HA_ID=0123456789
ovsdb-server --pidfile --detach --log-file --remote=punix:/var/run/openvswitch/db.sock --remote=db:hardware_vtep,Global,managers --remote=ptcp:6632 /etc/openvswitch/ovs.db /etc/openvswitch/vtep.db
ovs-vswitchd --log-file --detach --pidfile unix:/var/run/openvswitch/db.sock
ovs-vsctl add-br $BRIDGE
ovs-vsctl add-port $BRIDGE eth2
vtep-ctl add-ps $BRIDGE
vtep-ctl set Physical_Switch $BRIDGE tunnel_ips=$vtep
/usr/share/openvswitch/scripts/ovs-vtep --log-file=/var/log/openvswitch/ovs-vtep.log --pidfile=/var/run/openvswitch/ovs-vtep.pid --detach $BRIDGE
vtep-ctl set-manager tcp:172.16.1.30:6640
MANAGERUUID=$(vtep-ctl list Manager | grep ^_uuid | awk -F: '

{print $2}

' | tr -d ' ')
vtep-ctl set Manager $MANAGERUUID other_config:ha_enabled=true
vtep-ctl set Manager $MANAGERUUID other_config:ha_id=$HA_ID
vtep-ctl list Manager
---------------------------

Comment by Ran Xiao [ 19/Jun/17 ]

Please ignore this comment as we found this's a different bug.
We'll issue a new bug ticket for it later.
Sorry for the inconvenience.

(In reply to Ran Xiao from comment #1)
> UPDATES:
>
> The following error occurred when we adding HWVTEP Node.
> It seems to be the same cause.
>
> Errors:
> 2017-06-14 14:14:35,554 | ERROR | on-dispatcher-66 | TransactInvokerImpl
> | 292 - org.opendaylight.ovsdb.hwvtepsouthbound-impl - 1.4.1.SNAPSHOT |
> HWVTEP transaction operation failed referential integrity violation Table
> Physical_Port column vlan_bindings row d400dc78-d0b2-4a45-beba-326b18eb4d59
> references nonexistent row cc7cbc81-eb4d-4f2b-ba04-9dfa6afe849d in table
> Logical_Switch.
> 2017-06-14 14:14:35,555 | ERROR | on-dispatcher-66 | TransactInvokerImpl
> | 292 - org.opendaylight.ovsdb.hwvtepsouthbound-impl - 1.4.1.SNAPSHOT |
> Failed transaction update [tunnel_ips : [] , ]update [vlan_bindings :
>

{2222=LogicalSwitch_30ae2d8c_5361_45be_b7a8_a93abf722e8f} , ]
> 2017-06-14 14:14:36,047 | ERROR | on-dispatcher-66 | TransactInvokerImpl
> | 292 - org.opendaylight.ovsdb.hwvtepsouthbound-impl - 1.4.1.SNAPSHOT |
> HWVTEP transaction operation failed referential integrity violation Table
> Ucast_Macs_Remote column locator row 17ddb12a-f4de-487d-908f-1717790100f1
> references nonexistent row 00000000-0000-0000-0000-000000000000 in table
> Physical_Locator.
> 2017-06-14 14:14:36,047 | ERROR | on-dispatcher-66 | TransactInvokerImpl
> | 292 - org.opendaylight.ovsdb.hwvtepsouthbound-impl - 1.4.1.SNAPSHOT |
> Failed transaction insert [name : 30ae2d8c-5361-45be-b7a8-a93abf722e8f ,
> tunnel_key : [96] , ] insert [encapsulation_type : vxlan_over_ipv4 ,
> dst_ip : 10.0.0.10 , ] insert [locators :
> [PhysicalLocator_Random_61098e5741e640c6a118896b0c87e1c9] , ] insert
> [locator_set : PhysicalLocatorSet_Random_49a5353230ee467c8a8254689b15a162 ,
> logical_switch : LogicalSwitch_30ae2d8c_5361_45be_b7a8_a93abf722e8f , MAC :
> unknown-dst , ] insert [logical_switch :
> LogicalSwitch_30ae2d8c_5361_45be_b7a8_a93abf722e8f , MAC : fa:16:3e:fa:1d:10
> , ]
>
> Cause of failure:
> The operation order of OVSDB is incorrect.
>
> The uuid of LogicalSwitch_30ae2d8c_5361_45be_b7a8_a93abf722e8f will not be
> assigned if update [vlan_bindings :
> {2222=LogicalSwitch_30ae2d8c_5361_45be_b7a8_a93abf722e8f}

, ] is not
> executed after Logical Switch creation with operation of insert [name :
> 30ae2d8c-5361-45be-b7a8-a93abf722e8f , tunnel_key : [96] , ]
>
> Root cause:
> The order of OVSDB operations are not considered
>
>
> Detailed info of adding HWVTEP Node
> Steps executed:
> 1. In HWVTEP#1
> Create GATEWAY and CONNECTION for this HWVTEP.
> Confirmed the communication from DHCP namespace to Baremetal is OK.
> 2. In new node, HWVTEP#2
> Start up with the same parameters as HWVTEP#1 using.
> And set up HA Cluster.
> For 'other_config:ha_id', 'vtep', 'Physical Switch name', we specified
> the same value for HWVTEP#1 and #2.
> 'other_config:ha_enabled' is set to 'true'
>
> Commans executed:
> ---- HWVTEP#1/HWVTEP#2 ----
> vtep=10.0.0.50
> BRIDGE=ocata-l2gw1
> HA_ID=0123456789
> ovsdb-server --pidfile --detach --log-file
> --remote=punix:/var/run/openvswitch/db.sock
> --remote=db:hardware_vtep,Global,managers --remote=ptcp:6632
> /etc/openvswitch/ovs.db /etc/openvswitch/vtep.db
> ovs-vswitchd --log-file --detach --pidfile unix:/var/run/openvswitch/db.sock
> ovs-vsctl add-br $BRIDGE
> ovs-vsctl add-port $BRIDGE eth2
> vtep-ctl add-ps $BRIDGE
> vtep-ctl set Physical_Switch $BRIDGE tunnel_ips=$vtep
> /usr/share/openvswitch/scripts/ovs-vtep
> --log-file=/var/log/openvswitch/ovs-vtep.log
> --pidfile=/var/run/openvswitch/ovs-vtep.pid --detach $BRIDGE
> vtep-ctl set-manager tcp:172.16.1.30:6640
> MANAGERUUID=$(vtep-ctl list Manager | grep ^_uuid | awk -F: '

{print $2}

' |
> tr -d ' ')
> vtep-ctl set Manager $MANAGERUUID other_config:ha_enabled=true
> vtep-ctl set Manager $MANAGERUUID other_config:ha_id=$HA_ID
> vtep-ctl list Manager
> ---------------------------

Comment by Ran Xiao [ 22/Jun/17 ]

This bug will be fixed by the proposal of OVSDB-418.
After the code are fixed and committed, I will close this bug.

Comment by suneel verma [ 03/Jul/17 ]

Ran,
Can you verify if it is still the issue ? The other issue OVSDB-418 is closed.

Comment by Ran Xiao [ 04/Jul/17 ]

Suneel,
Thanks.
I will close this ticket as OVSDB-418 is closed.

Generated at Wed Feb 07 20:36:21 UTC 2024 using Jira 8.20.10#820010-sha1:ace47f9899e9ee25d7157d59aa17ab06aee30d3d.