[OVSDB-242] GatewayMacResolverService continues to try to resolve gateway after nodes have disconnected. Created: 27/Dec/15  Updated: 26/Jan/16  Resolved: 26/Jan/16

Status: Resolved
Project: ovsdb
Component/s: openstack.net-virt
Affects Version/s: unspecified
Fix Version/s: None

Type: Bug
Reporter: Andre Fredette Assignee: Flavio Fernandes
Resolution: Cannot Reproduce Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Operating System: All
Platform: All


Issue Links:
Duplicate
is duplicated by OVSDB-199 Externalgateway Mac address keeps on ... Resolved
is duplicated by OVSDB-214 Unreliable ERROR message as Unable to... Resolved
External issue ID: 4844

 Description   

I have the 2-node setup described here: https://docs.google.com/presentation/d/1KIuNDuUJGGEV37Zk9yzx9OSnWExt4iD2Z7afycFLf_I/edit#slide=id.p4

Except, I'm using an external ODL controller built from the master branch (I've seen this with the Li branch also).

After I unstack the nodes, the GatewayMacResolverService (aka periodic ARP) continues to try to resolve gateway continues to try to resolve the MAC for the gateway as indicated by the following error in the log that keeps repeating:

2015-12-27 07:51:01,704 | ERROR | pool-48-thread-1 | GatewayMacResolverService | 269 - org.opendaylight.ovsdb.openstack.net-virt-providers - 1.2.1.SNAPSHOT | MAC address for gateway 192.168.56.1 can not be resolved, because external bridge 6822471333697 is not connected to controller.

Full log can be seen here:
https://gist.github.com/0b6de0de84ff65d3887c

Can we recognize this case, and delete the gateway?



 Comments   
Comment by Flavio Fernandes [ 08/Jan/16 ]

I did a little experiment using stable lithium and this is possibly an issue in Be, but I'm not sure.

it seems that the neutron port is not getting deleted as far as odl goes and that is what odl needs in order to stop the periodic mac resolver. So possibly an issue in neutron northbound.

vagrant@devstack-control:~/devstack$ cd /home/vagrant/devstack/
vagrant@devstack-control:~/devstack$
vagrant@devstack-control:~/devstack$ source openrc admin admin
vagrant@devstack-control:~/devstack$ keystone user-create --name=user1 --pass=user1 --email=user1@example.com 2> /dev/null
vagrant@devstack-control:~/devstack$ keystone user-role-add --user=user1 --role=member --tenant=tenant1 2> /dev/null
vagrant@devstack-control:~/devstack$
vagrant@devstack-control:~/devstack$ TNT1_ID=$(keystone tenant-list 2> /dev/null | grep '\s'tenant1'' | awk '

{print $2}

')
vagrant@devstack-control:~/devstack$
vagrant@devstack-control:~/devstack$ ## neutron net-create ext --shared --provider:network_type flat --router:external --provider:physical_network physnetext
vagrant@devstack-control:~/devstack$ neutron net-create ext --provider:network_type flat --router:external --provider:physical_network physnetext
Created a new network:
---------------------------------------------------------------+

Field Value

---------------------------------------------------------------+

admin_state_up True
id 0ec9caf4-ee01-477b-98f5-691047247aa0
mtu 0
name ext
port_security_enabled True
provider:network_type flat
provider:physical_network physnetext
provider:segmentation_id  
router:external True
shared False
status ACTIVE
subnets  
tenant_id 8fa577a7028e4ac69246e1edc814b35e

---------------------------------------------------------------+
vagrant@devstack-control:~/devstack$
vagrant@devstack-control:~/devstack$ neutron subnet-create \
> --allocation-pool start=192.168.111.21,end=192.168.111.40 --gateway=192.168.111.254 \
> --disable-dhcp --name subext ext 192.168.111.0/24
Created a new subnet:
-----------------------------------------------------------------------+

Field Value

-----------------------------------------------------------------------+

allocation_pools {"start": "192.168.111.21", "end": "192.168.111.40"}
cidr 192.168.111.0/24
dns_nameservers  
enable_dhcp False
gateway_ip 192.168.111.254
host_routes  
id f4c8fcf4-4d6f-4535-975f-3bbbd9487b0c
ip_version 4
ipv6_address_mode  
ipv6_ra_mode  
name subext
network_id 0ec9caf4-ee01-477b-98f5-691047247aa0
subnetpool_id  
tenant_id 8fa577a7028e4ac69246e1edc814b35e

-----------------------------------------------------------------------+
vagrant@devstack-control:~/devstack$
vagrant@devstack-control:~/devstack$
vagrant@devstack-control:~/devstack$ unset SERVICE_TOKEN SERVICE_ENDPOINT
vagrant@devstack-control:~/devstack$ export OS_AUTH_URL=http://192.168.50.20:35357/v2.0/
vagrant@devstack-control:~/devstack$ export OS_REGION_NAME=RegionOne
vagrant@devstack-control:~/devstack$
vagrant@devstack-control:~/devstack$ ####=================
vagrant@devstack-control:~/devstack$
vagrant@devstack-control:~/devstack$ export OS_USERNAME=user1
vagrant@devstack-control:~/devstack$ export OS_TENANT_NAME=tenant1
vagrant@devstack-control:~/devstack$ export OS_PASSWORD=user1
vagrant@devstack-control:~/devstack$ export PS1='[\u@\h \W(keystone_user1)]\$ '
[vagrant@devstack-control devstack(keystone_user1)]$
[vagrant@devstack-control devstack(keystone_user1)]$ rm -f id_rsa_demo* ; ssh-keygen -t rsa -b 2048 -N '' -f id_rsa_demo > /dev/null
[vagrant@devstack-control devstack(keystone_user1)]$ nova keypair-add --pub-key id_rsa_demo.pub demo_key > /dev/null
[vagrant@devstack-control devstack(keystone_user1)]$
[vagrant@devstack-control devstack(keystone_user1)]$ neutron router-create rtr1
Created a new router:
-----------------------------------------------------------+

Field Value

-----------------------------------------------------------+

admin_state_up True
external_gateway_info  
id 8b28cdc9-2b32-429e-b08b-f61e19b29057
name rtr1
routes  
status ACTIVE
tenant_id 68c40cdb8e034a1f8546a9411c4fc0e5

-----------------------------------------------------------+
[vagrant@devstack-control devstack(keystone_user1)]$
[vagrant@devstack-control devstack(keystone_user1)]$
[vagrant@devstack-control devstack(keystone_user1)]$ neutron router-gateway-set rtr1 ext
Set gateway for router rtr1
[vagrant@devstack-control devstack(keystone_user1)]$
[vagrant@devstack-control devstack(keystone_user1)]$
[vagrant@devstack-control devstack(keystone_user1)]$
[vagrant@devstack-control devstack(keystone_user1)]$ neutron router-gateway-clear rtr1
Removed gateway from router rtr1
[vagrant@devstack-control devstack(keystone_user1)]$
[vagrant@devstack-control devstack(keystone_user1)]$
[vagrant@devstack-control devstack(keystone_user1)]$
[vagrant@devstack-control devstack(keystone_user1)]$ neutron router-delete rtr1
Deleted router: rtr1

[vagrant@devstack-control devstack(keystone_user1)]$ curl -u admin:admin http://192.168.50.1:8080/controller/nb/v2/neutron/ports
{
"ports" : [ {
"id" : "0f8f35f1-5248-417a-a5d1-ad180049fca4",
"network_id" : "0ec9caf4-ee01-477b-98f5-691047247aa0",
"name" : "",
"admin_state_up" : true,
"mac_address" : "FA:16:3E:8F:35:28",
"fixed_ips" : [

{ "ip_address" : "192.168.111.21", "subnet_id" : "f4c8fcf4-4d6f-4535-975f-3bbbd9487b0c" }

],
"device_id" : "8b28cdc9-2b32-429e-b08b-f61e19b29057",
"device_owner" : "network:router_gateway",
"tenant_id" : "8fa577a7028e4ac69246e1edc814b35e",
"security_groups" : [ ],
"allowed_address_pairs" : [ ],
"binding:host_id" : "",
"binding:vnic_type" : "normal",
"binding:vif_type" : "unbound",
"binding:vif_details" : [ {
} ]
} ]
}[vagrant@devstack-control devstack(keystone_user1)]$ exit
logout
Connection to 127.0.0.1 closed.

[Flavios-MBP:devstack-nodes-liberty.git] (liberty)$ vagrant ssh
vagrant@devstack-control:~$ cd devstack/
vagrant@devstack-control:~/devstack$ source openrc admin admin
vagrant@devstack-control:~/devstack$ neutron subnet-list
----------------------------------------------------------------------------------------------------------------+

id name cidr allocation_pools

----------------------------------------------------------------------------------------------------------------+

f4c8fcf4-4d6f-4535-975f-3bbbd9487b0c subext 192.168.111.0/24 {"start": "192.168.111.21", "end": "192.168.111.40"}

----------------------------------------------------------------------------------------------------------------+
vagrant@devstack-control:~/devstack$ neutron subnet-delete subext
Deleted subnet: subext

vagrant@devstack-control:~/devstack$ curl -u admin:admin http://192.168.50.1:8080/controller/nb/v2/neutron/ports
{
"ports" : [ {
"id" : "0f8f35f1-5248-417a-a5d1-ad180049fca4",
"network_id" : "0ec9caf4-ee01-477b-98f5-691047247aa0",
"name" : "",
"admin_state_up" : true,
"mac_address" : "FA:16:3E:8F:35:28",
"fixed_ips" : [

{ "ip_address" : "192.168.111.21", "subnet_id" : "f4c8fcf4-4d6f-4535-975f-3bbbd9487b0c" }

],
"device_id" : "8b28cdc9-2b32-429e-b08b-f61e19b29057",
"device_owner" : "network:router_gateway",
"tenant_id" : "8fa577a7028e4ac69246e1edc814b35e",
"security_groups" : [ ],
"allowed_address_pairs" : [ ],
"binding:host_id" : "",
"binding:vnic_type" : "normal",
"binding:vif_type" : "unbound",
"binding:vif_details" : [ {
} ]
} ]
}
vagrant@devstack-control:~/devstack$ neutron port-list

vagrant@devstack-control:~/devstack$

Comment by Flavio Fernandes [ 26/Jan/16 ]

I tested this using today's (Jan/26/16) Beryllium and Lithium and have convinced myself that this is working
as expected. The critical event that needs to happen is the neutron port delete. Once that happens, the following
code path takes place:

PortHandler::doNeutronPortDeleted() ->
NeutronL3Adapter::handleNeutronPortEvent() ->
GatewayMacResolverService::stopPeriodicRefresh() ->
GatewayMacResolverService::resetFlowToRemove()

There is another issue at play here which makes this a lower priority: unstack vs. neutron port clear.
To make the code path mentioned above happen, I did these commands:

karaf> log:set debug org.opendaylight.ovsdb.openstack.netvirt.providers.openflow13.services.arp.GatewayMacResolverService

https://gist.github.com/6848cb3941271cc159ba

source openrc admin admin
neutron net-create ext --provider:network_type flat --router:external --provider:physical_network physnetext

neutron subnet-create --allocation-pool start=192.168.111.21,end=192.168.111.40 --gateway=192.168.111.254 \
--disable-dhcp --name subext ext 192.168.111.0/24

neutron router-create rtr1
neutron router-gateway-set rtr1 ext

  1. neutron ports known to ODL are shown here: https://gist.github.com/db246ca1e4daed2dca4c
  1. to remove neutron port used as router interface
    neutron router-gateway-clear rtr1
  1. neutron ports are indeed removed after the command above
    vagrant@devstack-control:~$ curl -u admin:admin http://192.168.50.1:8080/controller/nb/v2/neutron/ports { "ports" : [ ] }

output when I ran test against Lithium: https://gist.github.com/13824b02224ec343765f

karaf log when neutron port is removed shows the timer being stopped: https://gist.github.com/afe90432d8d8144dee0c

That is different than ./unstack, which makes no attempt in clearing the config in ODL.
The reconcile code would remove this port once neutron re-connects to ODL, but short of that
happening, there is not much ODL can do to differentiate unstack from a temporary disconnect
in the networking-odl side.

Last point to make here is that ODL gateway resolver code reacts to neutron callbacks, not OVS. That is because mac resolver is made to be resilient enough to change br-ex every 10 seconds until it succeeds.

Generated at Wed Feb 07 20:35:52 UTC 2024 using Jira 8.20.10#820010-sha1:ace47f9899e9ee25d7157d59aa17ab06aee30d3d.