https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/OVSDB-242 ovsdb <p>I have the 2-node setup described here: <a href="https://docs.google.com/presentation/d/1KIuNDuUJGGEV37Zk9yzx9OSnWExt4iD2Z7afycFLf_I/edit#slide=id.p4" class="external-link" target="_blank" rel="nofollow noopener">https://docs.google.com/presentation/d/1KIuNDuUJGGEV37Zk9yzx9OSnWExt4iD2Z7afycFLf_I/edit#slide=id.p4</a></p> <p>Except, I'm using an external ODL controller built from the master branch (I've seen this with the Li branch also).</p> <p>After I unstack the nodes, the GatewayMacResolverService (aka periodic ARP) continues to try to resolve gateway continues to try to resolve the MAC for the gateway as indicated by the following error in the log that keeps repeating:</p> <p>2015-12-27 07:51:01,704 | ERROR | pool-48-thread-1 | GatewayMacResolverService | 269 - org.opendaylight.ovsdb.openstack.net-virt-providers - 1.2.1.SNAPSHOT | MAC address for gateway 192.168.56.1 can not be resolved, because external bridge 6822471333697 is not connected to controller.</p> <p>Full log can be seen here:<br/> <a href="https://gist.github.com/0b6de0de84ff65d3887c" class="external-link" target="_blank" rel="nofollow noopener">https://gist.github.com/0b6de0de84ff65d3887c</a></p> <p>Can we recognize this case, and delete the gateway?</p> <p>Operating System: All<br/> Platform: All</p> OVSDB-242

GatewayMacResolverService continues to try to resolve gateway after nodes have disconnected.

Bug Resolved Cannot Reproduce Flavio Fernandes Andre Fredette Sun, 27 Dec 2015 13:08:09 +0000 Tue, 26 Jan 2016 21:40:51 +0000 Tue, 26 Jan 2016 21:40:51 +0000 unspecified openstack.net-virt 0 3 <p>I did a little experiment using stable lithium and this is possibly an issue in Be, but I'm not sure.</p> <p>it seems that the neutron port is not getting deleted as far as odl goes and that is what odl needs in order to stop the periodic mac resolver. So possibly an issue in neutron northbound.</p> <p>&#8211;</p> <p>vagrant@devstack-control:~/devstack$ cd /home/vagrant/devstack/<br/> vagrant@devstack-control:~/devstack$ <br/> vagrant@devstack-control:~/devstack$ source openrc admin admin<br/> vagrant@devstack-control:~/devstack$ keystone user-create --name=user1 --pass=user1 --email=user1@example.com 2&gt; /dev/null<br/> vagrant@devstack-control:~/devstack$ keystone user-role-add --user=user1 --role=<em>member</em> --tenant=tenant1 2&gt; /dev/null<br/> vagrant@devstack-control:~/devstack$ <br/> vagrant@devstack-control:~/devstack$ TNT1_ID=$(keystone tenant-list 2&gt; /dev/null | grep '\s'tenant1'' | awk '</p> {print $2} <p>')<br/> vagrant@devstack-control:~/devstack$ <br/> vagrant@devstack-control:~/devstack$ ## neutron net-create ext --shared --provider:network_type flat --router:external --provider:physical_network physnetext<br/> vagrant@devstack-control:~/devstack$ neutron net-create ext --provider:network_type flat --router:external --provider:physical_network physnetext<br/> Created a new network:<br/> <ins>--------------------------<del></ins></del>-------------------------------------+</p> <div class='table-wrap'> <table class='confluenceTable'><tbody> <tr> <td class='confluenceTd'> Field </td> <td class='confluenceTd'> Value </td> </tr> </tbody></table> </div> <p><ins>--------------------------<del></ins></del>-------------------------------------+</p> <div class='table-wrap'> <table class='confluenceTable'><tbody> <tr> <td class='confluenceTd'> admin_state_up </td> <td class='confluenceTd'> True </td> </tr> <tr> <td class='confluenceTd'> id </td> <td class='confluenceTd'> 0ec9caf4-ee01-477b-98f5-691047247aa0 </td> </tr> <tr> <td class='confluenceTd'> mtu </td> <td class='confluenceTd'> 0 </td> </tr> <tr> <td class='confluenceTd'> name </td> <td class='confluenceTd'> ext </td> </tr> <tr> <td class='confluenceTd'> port_security_enabled </td> <td class='confluenceTd'> True </td> </tr> <tr> <td class='confluenceTd'> provider:network_type </td> <td class='confluenceTd'> flat </td> </tr> <tr> <td class='confluenceTd'> provider:physical_network </td> <td class='confluenceTd'> physnetext </td> </tr> <tr> <td class='confluenceTd'> provider:segmentation_id </td> <td class='confluenceTd'>&nbsp;</td> </tr> <tr> <td class='confluenceTd'> router:external </td> <td class='confluenceTd'> True </td> </tr> <tr> <td class='confluenceTd'> shared </td> <td class='confluenceTd'> False </td> </tr> <tr> <td class='confluenceTd'> status </td> <td class='confluenceTd'> ACTIVE </td> </tr> <tr> <td class='confluenceTd'> subnets </td> <td class='confluenceTd'>&nbsp;</td> </tr> <tr> <td class='confluenceTd'> tenant_id </td> <td class='confluenceTd'> 8fa577a7028e4ac69246e1edc814b35e </td> </tr> </tbody></table> </div> <p><ins>--------------------------<del></ins></del>-------------------------------------+<br/> vagrant@devstack-control:~/devstack$ <br/> vagrant@devstack-control:~/devstack$ neutron subnet-create \<br/> &gt; --allocation-pool start=192.168.111.21,end=192.168.111.40 --gateway=192.168.111.254 \<br/> &gt; --disable-dhcp --name subext ext 192.168.111.0/24<br/> Created a new subnet:<br/> <ins>------------------<del></ins></del>-----------------------------------------------------+</p> <div class='table-wrap'> <table class='confluenceTable'><tbody> <tr> <td class='confluenceTd'> Field </td> <td class='confluenceTd'> Value </td> </tr> </tbody></table> </div> <p><ins>------------------<del></ins></del>-----------------------------------------------------+</p> <div class='table-wrap'> <table class='confluenceTable'><tbody> <tr> <td class='confluenceTd'> allocation_pools </td> <td class='confluenceTd'> {"start": "192.168.111.21", "end": "192.168.111.40"} </td> </tr> <tr> <td class='confluenceTd'> cidr </td> <td class='confluenceTd'> 192.168.111.0/24 </td> </tr> <tr> <td class='confluenceTd'> dns_nameservers </td> <td class='confluenceTd'>&nbsp;</td> </tr> <tr> <td class='confluenceTd'> enable_dhcp </td> <td class='confluenceTd'> False </td> </tr> <tr> <td class='confluenceTd'> gateway_ip </td> <td class='confluenceTd'> 192.168.111.254 </td> </tr> <tr> <td class='confluenceTd'> host_routes </td> <td class='confluenceTd'>&nbsp;</td> </tr> <tr> <td class='confluenceTd'> id </td> <td class='confluenceTd'> f4c8fcf4-4d6f-4535-975f-3bbbd9487b0c </td> </tr> <tr> <td class='confluenceTd'> ip_version </td> <td class='confluenceTd'> 4 </td> </tr> <tr> <td class='confluenceTd'> ipv6_address_mode </td> <td class='confluenceTd'>&nbsp;</td> </tr> <tr> <td class='confluenceTd'> ipv6_ra_mode </td> <td class='confluenceTd'>&nbsp;</td> </tr> <tr> <td class='confluenceTd'> name </td> <td class='confluenceTd'> subext </td> </tr> <tr> <td class='confluenceTd'> network_id </td> <td class='confluenceTd'> 0ec9caf4-ee01-477b-98f5-691047247aa0 </td> </tr> <tr> <td class='confluenceTd'> subnetpool_id </td> <td class='confluenceTd'>&nbsp;</td> </tr> <tr> <td class='confluenceTd'> tenant_id </td> <td class='confluenceTd'> 8fa577a7028e4ac69246e1edc814b35e </td> </tr> </tbody></table> </div> <p><ins>------------------<del></ins></del>-----------------------------------------------------+<br/> vagrant@devstack-control:~/devstack$ <br/> vagrant@devstack-control:~/devstack$ <br/> vagrant@devstack-control:~/devstack$ unset SERVICE_TOKEN SERVICE_ENDPOINT<br/> vagrant@devstack-control:~/devstack$ export OS_AUTH_URL=<a href="http://192.168.50.20:35357/v2.0/" class="external-link" target="_blank" rel="nofollow noopener">http://192.168.50.20:35357/v2.0/</a><br/> vagrant@devstack-control:~/devstack$ export OS_REGION_NAME=RegionOne<br/> vagrant@devstack-control:~/devstack$ <br/> vagrant@devstack-control:~/devstack$ ####=================<br/> vagrant@devstack-control:~/devstack$ <br/> vagrant@devstack-control:~/devstack$ export OS_USERNAME=user1<br/> vagrant@devstack-control:~/devstack$ export OS_TENANT_NAME=tenant1<br/> vagrant@devstack-control:~/devstack$ export OS_PASSWORD=user1<br/> vagrant@devstack-control:~/devstack$ export PS1='<span class="error">&#91;\u@\h \W(keystone_user1)&#93;</span>\$ '<br/> <span class="error">&#91;vagrant@devstack-control devstack(keystone_user1)&#93;</span>$ <br/> <span class="error">&#91;vagrant@devstack-control devstack(keystone_user1)&#93;</span>$ rm -f id_rsa_demo* ; ssh-keygen -t rsa -b 2048 -N '' -f id_rsa_demo &gt; /dev/null<br/> <span class="error">&#91;vagrant@devstack-control devstack(keystone_user1)&#93;</span>$ nova keypair-add --pub-key id_rsa_demo.pub demo_key &gt; /dev/null<br/> <span class="error">&#91;vagrant@devstack-control devstack(keystone_user1)&#93;</span>$ <br/> <span class="error">&#91;vagrant@devstack-control devstack(keystone_user1)&#93;</span>$ neutron router-create rtr1<br/> Created a new router:<br/> <ins>----------------------<del></ins></del>-------------------------------------+</p> <div class='table-wrap'> <table class='confluenceTable'><tbody> <tr> <td class='confluenceTd'> Field </td> <td class='confluenceTd'> Value </td> </tr> </tbody></table> </div> <p><ins>----------------------<del></ins></del>-------------------------------------+</p> <div class='table-wrap'> <table class='confluenceTable'><tbody> <tr> <td class='confluenceTd'> admin_state_up </td> <td class='confluenceTd'> True </td> </tr> <tr> <td class='confluenceTd'> external_gateway_info </td> <td class='confluenceTd'>&nbsp;</td> </tr> <tr> <td class='confluenceTd'> id </td> <td class='confluenceTd'> 8b28cdc9-2b32-429e-b08b-f61e19b29057 </td> </tr> <tr> <td class='confluenceTd'> name </td> <td class='confluenceTd'> rtr1 </td> </tr> <tr> <td class='confluenceTd'> routes </td> <td class='confluenceTd'>&nbsp;</td> </tr> <tr> <td class='confluenceTd'> status </td> <td class='confluenceTd'> ACTIVE </td> </tr> <tr> <td class='confluenceTd'> tenant_id </td> <td class='confluenceTd'> 68c40cdb8e034a1f8546a9411c4fc0e5 </td> </tr> </tbody></table> </div> <p><ins>----------------------<del></ins></del>-------------------------------------+<br/> <span class="error">&#91;vagrant@devstack-control devstack(keystone_user1)&#93;</span>$ <br/> <span class="error">&#91;vagrant@devstack-control devstack(keystone_user1)&#93;</span>$ <br/> <span class="error">&#91;vagrant@devstack-control devstack(keystone_user1)&#93;</span>$ neutron router-gateway-set rtr1 ext<br/> Set gateway for router rtr1<br/> <span class="error">&#91;vagrant@devstack-control devstack(keystone_user1)&#93;</span>$ <br/> <span class="error">&#91;vagrant@devstack-control devstack(keystone_user1)&#93;</span>$ <br/> <span class="error">&#91;vagrant@devstack-control devstack(keystone_user1)&#93;</span>$ <br/> <span class="error">&#91;vagrant@devstack-control devstack(keystone_user1)&#93;</span>$ neutron router-gateway-clear rtr1 <br/> Removed gateway from router rtr1<br/> <span class="error">&#91;vagrant@devstack-control devstack(keystone_user1)&#93;</span>$ <br/> <span class="error">&#91;vagrant@devstack-control devstack(keystone_user1)&#93;</span>$ <br/> <span class="error">&#91;vagrant@devstack-control devstack(keystone_user1)&#93;</span>$ <br/> <span class="error">&#91;vagrant@devstack-control devstack(keystone_user1)&#93;</span>$ neutron router-delete rtr1<br/> Deleted router: rtr1</p> <p><span class="error">&#91;vagrant@devstack-control devstack(keystone_user1)&#93;</span>$ curl -u admin:admin <a href="http://192.168.50.1:8080/controller/nb/v2/neutron/ports" class="external-link" target="_blank" rel="nofollow noopener">http://192.168.50.1:8080/controller/nb/v2/neutron/ports</a> <br/> {<br/> "ports" : [ {<br/> "id" : "0f8f35f1-5248-417a-a5d1-ad180049fca4",<br/> "network_id" : "0ec9caf4-ee01-477b-98f5-691047247aa0",<br/> "name" : "",<br/> "admin_state_up" : true,<br/> "mac_address" : "FA:16:3E:8F:35:28",<br/> "fixed_ips" : [ </p> { "ip_address" : "192.168.111.21", "subnet_id" : "f4c8fcf4-4d6f-4535-975f-3bbbd9487b0c" } <p> ],<br/> "device_id" : "8b28cdc9-2b32-429e-b08b-f61e19b29057",<br/> "device_owner" : "network:router_gateway",<br/> "tenant_id" : "8fa577a7028e4ac69246e1edc814b35e",<br/> "security_groups" : [ ],<br/> "allowed_address_pairs" : [ ],<br/> "binding:host_id" : "",<br/> "binding:vnic_type" : "normal",<br/> "binding:vif_type" : "unbound",<br/> "binding:vif_details" : [ {<br/> } ]<br/> } ]<br/> }<span class="error">&#91;vagrant@devstack-control devstack(keystone_user1)&#93;</span>$ exit<br/> logout<br/> Connection to 127.0.0.1 closed.</p> <p><span class="error">&#91;Flavios-MBP:devstack-nodes-liberty.git&#93;</span> (liberty)$ vagrant ssh <br/> vagrant@devstack-control:~$ cd devstack/<br/> vagrant@devstack-control:~/devstack$ source openrc admin admin<br/> vagrant@devstack-control:~/devstack$ neutron subnet-list<br/> <ins>-------------------------------------<del></ins></del>------<del><ins></del>----------------<del></ins></del>-----------------------------------------------------+</p> <div class='table-wrap'> <table class='confluenceTable'><tbody> <tr> <td class='confluenceTd'> id </td> <td class='confluenceTd'> name </td> <td class='confluenceTd'> cidr </td> <td class='confluenceTd'> allocation_pools </td> </tr> </tbody></table> </div> <p><ins>-------------------------------------<del></ins></del>------<del><ins></del>----------------<del></ins></del>-----------------------------------------------------+</p> <div class='table-wrap'> <table class='confluenceTable'><tbody> <tr> <td class='confluenceTd'> f4c8fcf4-4d6f-4535-975f-3bbbd9487b0c </td> <td class='confluenceTd'> subext </td> <td class='confluenceTd'> 192.168.111.0/24 </td> <td class='confluenceTd'> {"start": "192.168.111.21", "end": "192.168.111.40"} </td> </tr> </tbody></table> </div> <p><ins>-------------------------------------<del></ins></del>------<del><ins></del>----------------<del></ins></del>-----------------------------------------------------+<br/> vagrant@devstack-control:~/devstack$ neutron subnet-delete subext<br/> Deleted subnet: subext</p> <p>vagrant@devstack-control:~/devstack$ curl -u admin:admin <a href="http://192.168.50.1:8080/controller/nb/v2/neutron/ports" class="external-link" target="_blank" rel="nofollow noopener">http://192.168.50.1:8080/controller/nb/v2/neutron/ports</a><br/> {<br/> "ports" : [ {<br/> "id" : "0f8f35f1-5248-417a-a5d1-ad180049fca4",<br/> "network_id" : "0ec9caf4-ee01-477b-98f5-691047247aa0",<br/> "name" : "",<br/> "admin_state_up" : true,<br/> "mac_address" : "FA:16:3E:8F:35:28",<br/> "fixed_ips" : [ </p> { "ip_address" : "192.168.111.21", "subnet_id" : "f4c8fcf4-4d6f-4535-975f-3bbbd9487b0c" } <p> ],<br/> "device_id" : "8b28cdc9-2b32-429e-b08b-f61e19b29057",<br/> "device_owner" : "network:router_gateway",<br/> "tenant_id" : "8fa577a7028e4ac69246e1edc814b35e",<br/> "security_groups" : [ ],<br/> "allowed_address_pairs" : [ ],<br/> "binding:host_id" : "",<br/> "binding:vnic_type" : "normal",<br/> "binding:vif_type" : "unbound",<br/> "binding:vif_details" : [ {<br/> } ]<br/> } ]<br/> }<br/> vagrant@devstack-control:~/devstack$ neutron port-list</p> <p>vagrant@devstack-control:~/devstack$</p> <p>I tested this using today's (Jan/26/16) Beryllium and Lithium and have convinced myself that this is working<br/> as expected. The critical event that needs to happen is the neutron port delete. Once that happens, the following<br/> code path takes place:</p> <p>PortHandler::doNeutronPortDeleted() -&gt;<br/> NeutronL3Adapter::handleNeutronPortEvent() -&gt;<br/> GatewayMacResolverService::stopPeriodicRefresh() -&gt;<br/> GatewayMacResolverService::resetFlowToRemove()</p> <p>There is another issue at play here which makes this a lower priority: unstack vs. neutron port clear.<br/> To make the code path mentioned above happen, I did these commands:</p> <p> karaf&gt; log:set debug org.opendaylight.ovsdb.openstack.netvirt.providers.openflow13.services.arp.GatewayMacResolverService</p> <p> <a href="https://gist.github.com/6848cb3941271cc159ba" class="external-link" target="_blank" rel="nofollow noopener">https://gist.github.com/6848cb3941271cc159ba</a></p> <p> source openrc admin admin<br/> neutron net-create ext --provider:network_type flat --router:external --provider:physical_network physnetext</p> <p> neutron subnet-create --allocation-pool start=192.168.111.21,end=192.168.111.40 --gateway=192.168.111.254 \<br/> --disable-dhcp --name subext ext 192.168.111.0/24</p> <p> neutron router-create rtr1<br/> neutron router-gateway-set rtr1 ext</p> <ol> <li>neutron ports known to ODL are shown here: <a href="https://gist.github.com/db246ca1e4daed2dca4c" class="external-link" target="_blank" rel="nofollow noopener">https://gist.github.com/db246ca1e4daed2dca4c</a></li> </ol> <ol> <li>to remove neutron port used as router interface<br/> neutron router-gateway-clear rtr1</li> </ol> <ol> <li>neutron ports are indeed removed after the command above<br/> vagrant@devstack-control:~$ curl -u admin:admin <a href="http://192.168.50.1:8080/controller/nb/v2/neutron/ports" class="external-link" target="_blank" rel="nofollow noopener">http://192.168.50.1:8080/controller/nb/v2/neutron/ports</a> { "ports" : [ ] }</li> </ol> <p>output when I ran test against Lithium: <a href="https://gist.github.com/13824b02224ec343765f" class="external-link" target="_blank" rel="nofollow noopener">https://gist.github.com/13824b02224ec343765f</a></p> <p>karaf log when neutron port is removed shows the timer being stopped: <a href="https://gist.github.com/afe90432d8d8144dee0c" class="external-link" target="_blank" rel="nofollow noopener">https://gist.github.com/afe90432d8d8144dee0c</a></p> <p>That is different than ./unstack, which makes no attempt in clearing the config in ODL.<br/> The reconcile code would remove this port once neutron re-connects to ODL, but short of that<br/> happening, there is not much ODL can do to differentiate unstack from a temporary disconnect<br/> in the networking-odl side.</p> <p>Last point to make here is that ODL gateway resolver code reacts to neutron callbacks, not OVS. That is because mac resolver is made to be resilient enough to change br-ex every 10 seconds until it succeeds.</p> Duplicate OVSDB-199 OVSDB-214 Development External issue ID 4844 External issue URL Rank 0|i021iv: