[OVSDB-455] l2 agent marked down when no response to http://172.16.0.15:8081/restconf/operational/neutron:neutron/hostconfigs Created: 07/Mar/18  Updated: 01/May/19

Status: Open
Project: ovsdb
Component/s: Southbound.Open_vSwitch
Affects Version/s: Oxygen, Fluorine
Fix Version/s: Nitrogen-SR4, Fluorine-SR1

Type: Bug Priority: High
Reporter: Sam Hague Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: ds
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Duplicate
is duplicated by OVSDB-444 Port binding failure after rebooting ... Resolved
Relates
relates to NEUTRON-151 A large number of hostconfig related ... Open

 Description   

During scale testing the OVSDB connection is dropped and eventually leads to hostconfig gets to fail. Current theory is that when the connection is removed, operational is cleared and this impacts the neutron northbound hostconfig queries from networking-odl. networking-odl does a get operational restconf/operational/neutron:neutron/hostconfigs but neutron northbound does not return anything. At that point networking-odl marks the ODL l2-agent as down and does not try to use it anymore.

 Comments   
Comment by Sam Hague [ 07/Mar/18 ]

When running Browbeat+Rally scenario with netcreate-boot scenario (i.e., we spawn a VM and verify that it becomes ACTIVE) with concurrency set to 5 and times set to 2000, we observed that after spawning some 1422 VMs all subsequent VMs start to fail (i.e, they go into an ERROR state).
 
When we looked at the logs, we observed that after spawning around 1200+ VMs the errors started to creep in. The errors are mostly related to "ODL L2 agent (i.e., neutron-odlagent-portbinding)" becoming inactive and hence port-binding fails.
 
Error reported in Rally: Binding failed for port 000f35ae-9db5-4a9d-9324-255fe7c668d0, please check neutron logs for more information.', u'code': 500, u'created': u'2017-11-30T14:37:43Z'}

It looks like networking-odl marks "ODL L2" agent as down when the information is not returned as part of the following query 

http://172.16.0.15:8081/restconf/operational/neutron:neutron/hostconfigs

After the browbeat test completed, we could see that all the "ODL L2" agents were down/inactive.
On the compute nodes, we see that vswitchd connection to ODL Controller is flaky and this could be the reason why the agent info is not returned as part of hostconfig url.

   <SNIP>
   2017-11-30T17:40:12.968Z\|01049\|rconn\|INFO\|br-int<->tcp:172.16.0.17:6653: connected
   2017-11-30T17:40:12.969Z\|01050\|rconn\|INFO\|br-int<->tcp:172.16.0.17:6653: connection closed by peer
   2017-11-30T17:40:12.969Z\|01051\|rconn\|INFO\|br-int<->tcp:172.16.0.17:6653: continuing to retry connections in the background but suppressing further logging
   2017-11-30T17:40:22.778Z\|01052\|rconn\|INFO\|br-int<->tcp:172.16.0.17:6653: connected
   2017-11-30T17:40:22.778Z\|01053\|rconn\|INFO\|br-int<->tcp:172.16.0.17:6653: connection closed by peer
   <SNIP>

Setup:
OSP12 image
ODL running in a non-cluster, non-container setup with 1 controller and 32 compute nodes.
Port-status is disabled in this setup and inactivity_probe/max_backoff are appropriately configured.

RPMs used:
python-networking-odl-11.0.0-2.el7ost.noarch
opendaylight-6.2.0-5.el7ost.noarch

neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead.
+--------------------------------------+----------------+------------------------------------+-------------------+-------+----------------+------------------------------+
\| id                                   \| agent_type     \| host                               \| availability_zone \| alive \| admin_state_up \| binary                       \|
+--------------------------------------+----------------+------------------------------------+-------------------+-------+----------------+------------------------------+
\| 0201be96-3c80-451a-971e-dfdeb7ebbdd3 \| ODL L2         \| overcloud-compute-6.localdomain    \|                   \| xxx   \| True           \| neutron-odlagent-portbinding \|
\| 0283ed5d-b9c8-43c9-9c99-6c27b319b041 \| ODL L2         \| overcloud-compute-15.localdomain   \|                   \| xxx   \| True           \| neutron-odlagent-portbinding \|
\| 1ad29e22-cbf8-4384-b1e3-c14e3779a4ae \| ODL L2         \| overcloud-compute-18.localdomain   \|                   \| xxx   \| True           \| neutron-odlagent-portbinding \|
\| 29687ccb-aa71-4150-8059-75cdcc98b284 \| ODL L2         \| overcloud-compute-5.localdomain    \|                   \| xxx   \| True           \| neutron-odlagent-portbinding \|
\| 29b494c2-bfda-4cdc-b189-1713d66793b0 \| ODL L2         \| overcloud-compute-22.localdomain   \|                   \| xxx   \| True           \| neutron-odlagent-portbinding \|
\| 2e981caa-6d6e-4a0d-b716-d3f0d79c4968 \| ODL L2         \| overcloud-compute-12.localdomain   \|                   \| xxx   \| True           \| neutron-odlagent-portbinding \|
\| 3928ceb7-fc75-4309-a38d-2a8cad2b5666 \| ODL L2         \| overcloud-compute-14.localdomain   \|                   \| xxx   \| True           \| neutron-odlagent-portbinding \|
\| 4011168b-2bda-4bbc-8b3d-0458120b4c13 \| ODL L2         \| overcloud-compute-1.localdomain    \|                   \| xxx   \| True           \| neutron-odlagent-portbinding \|
\| 4404a197-73a2-41b7-a905-519a06318af0 \| ODL L2         \| overcloud-compute-30.localdomain   \|                   \| xxx   \| True           \| neutron-odlagent-portbinding \|
\| 4d148674-1e2f-4bcd-986a-505b8bc8f84f \| ODL L2         \| overcloud-compute-19.localdomain   \|                   \| xxx   \| True           \| neutron-odlagent-portbinding \|
\| 5623cb1c-0dcf-4126-b4f6-f19411e6ec5f \| ODL L2         \| overcloud-compute-26.localdomain   \|                   \| xxx   \| True           \| neutron-odlagent-portbinding \|
\| 5703d801-49dc-40d7-adda-e7949a6b6771 \| ODL L2         \| overcloud-compute-13.localdomain   \|                   \| xxx   \| True           \| neutron-odlagent-portbinding \|
\| 62d09361-4222-4a0a-8b91-d03dd324f1e7 \| ODL L2         \| overcloud-controller-0.localdomain \|                   \| xxx   \| True           \| neutron-odlagent-portbinding \|
\| 66edd958-936d-48c9-99e3-4d9241315fe7 \| ODL L2         \| overcloud-compute-28.localdomain   \|                   \| xxx   \| True           \| neutron-odlagent-portbinding \|
\| 682cfb7e-9cd0-4cf8-a1c9-ea8c6450ec37 \| ODL L2         \| overcloud-compute-8.localdomain    \|                   \| xxx   \| True           \| neutron-odlagent-portbinding \|
\| 72292f55-6a4a-44f2-85a1-ac3640f1b9f0 \| ODL L2         \| overcloud-compute-9.localdomain    \|                   \| xxx   \| True           \| neutron-odlagent-portbinding \|
\| 75ae43a1-2dd8-4223-ae34-400bbd3b5ac8 \| ODL L2         \| overcloud-compute-11.localdomain   \|                   \| xxx   \| True           \| neutron-odlagent-portbinding \|
\| 7816271e-9c9c-49a6-9cc4-b56c945fd92b \| ODL L2         \| overcloud-compute-4.localdomain    \|                   \| xxx   \| True           \| neutron-odlagent-portbinding \|
\| 7b567e37-5ca3-4e3d-91de-31ae56191a2a \| ODL L2         \| overcloud-compute-25.localdomain   \|                   \| xxx   \| True           \| neutron-odlagent-portbinding \|
\| 7c7f5637-4242-4015-ac1d-ce0932346519 \| ODL L2         \| overcloud-compute-21.localdomain   \|                   \| xxx   \| True           \| neutron-odlagent-portbinding \|
\| 7f7c54d8-028e-458c-8231-c7774b86e1cf \| ODL L2         \| overcloud-compute-3.localdomain    \|                   \| xxx   \| True           \| neutron-odlagent-portbinding \|
\| 9270a276-9e0c-42a9-85f5-513bcb9e717a \| ODL L2         \| overcloud-compute-2.localdomain    \|                   \| xxx   \| True           \| neutron-odlagent-portbinding \|
\| 92b3383d-0fd3-4578-94e9-3e67239b86a9 \| ODL L2         \| overcloud-compute-24.localdomain   \|                   \| xxx   \| True           \| neutron-odlagent-portbinding \|
\| 94cc6e57-0ffe-4243-ac1d-46f17cee5ed7 \| ODL L2         \| overcloud-compute-31.localdomain   \|                   \| xxx   \| True           \| neutron-odlagent-portbinding \|
\| 9fc823e4-146b-417d-9c13-078b11d6e7b9 \| ODL L2         \| overcloud-compute-20.localdomain   \|                   \| xxx   \| True           \| neutron-odlagent-portbinding \|
\| 9feaf78a-f2da-477d-8053-cc599b496aa8 \| ODL L2         \| overcloud-compute-27.localdomain   \|                   \| xxx   \| True           \| neutron-odlagent-portbinding \|
\| a2b758db-06b7-490e-b6d0-d05938848a23 \| ODL L2         \| overcloud-compute-16.localdomain   \|                   \| xxx   \| True           \| neutron-odlagent-portbinding \|
\| a68eb489-1ddd-4ab6-bbc9-931aa892282c \| ODL L2         \| overcloud-compute-17.localdomain   \|                   \| xxx   \| True           \| neutron-odlagent-portbinding \|
\| ace04e9f-a647-471f-92c9-e3f10e036fd7 \| ODL L2         \| overcloud-compute-10.localdomain   \|                   \| xxx   \| True           \| neutron-odlagent-portbinding \|
\| b5fc5492-edf0-4321-bf9f-41f1c541e381 \| ODL L2         \| overcloud-compute-0.localdomain    \|                   \| xxx   \| True           \| neutron-odlagent-portbinding \|
\| bef93822-1da4-48b6-b80e-16733f02c858 \| Metadata agent \| overcloud-controller-0.localdomain \|                   \| :-)   \| True           \| neutron-metadata-agent       \|
\| c085c9cd-412c-4a28-8280-d5521de0099c \| ODL L2         \| overcloud-compute-29.localdomain   \|                   \| xxx   \| True           \| neutron-odlagent-portbinding \|
\| f2fce8b9-7584-4255-ab7e-02475d82826f \| ODL L2         \| overcloud-compute-23.localdomain   \|                   \| xxx   \| True           \| neutron-odlagent-portbinding \|
\| fc35e8f7-8e39-49df-9824-2447367bd59f \| ODL L2         \| overcloud-compute-7.localdomain    \|                   \| xxx   \| True           \| neutron-odlagent-portbinding \|
\| fdf5a3be-d458-461b-8e94-e99ecdf5e435 \| DHCP agent     \| overcloud-controller-0.localdomain \| nova              \| :-)   \| True           \| neutron-dhcp-agent           \|
+--------------------------------------+----------------+------------------------------------+-------------------+-------+----------------+------------------------------+|
Generated at Wed Feb 07 20:36:27 UTC 2024 using Jira 8.20.10#820010-sha1:ace47f9899e9ee25d7157d59aa17ab06aee30d3d.