OpenDaylight JIRA

OpenDaylight JIRA https://jira.opendaylight.org This file is an XML representation of an issue en-us

8.20.10

820010

22-06-2022

[NEUTRON-204] networking-odl gives up on websocket, stops retrying https://jira.opendaylight.org/browse/NEUTRON-204 neutron <p>after bouncing all three nodes of an ODL cluster, websocket registration fails and networking-odl gives up. From that point forward, the netvirt workflow is broken</p> NEUTRON-204

networking-odl gives up on websocket, stops retrying

Bug High Open Unresolved Josh Hershberg Jamo Luhrsen csit:3node Thu, 8 Nov 2018 01:22:41 +0000 Thu, 10 Jan 2019 23:58:11 +0000 Fluorine-SR2 Neon northbound-api 0 2 <p>Is NEUTRON the right project for this?</p> <p>copy paste from an email:</p> <p>I think we are really close with everything, but I want to get to the bottom<br/> of another one. job is here <span class="error">[0]</span></p> <p>Essentially, what's happening is that networking-odl just quits trying to<br/> get a websocket. The final error is this:</p> <p>2018-11-07 19:35:45.116 sERROR networking_odl.common.websocket_client None req-4b2fa066-886c-4c1b-a836-8ea50cafa8ae None None websocket irrecoverable error </p> <p>which is from here <span class="error">[1]</span></p> <p>I think what's happening is that the /restconf call to make the registration<br/> gets a 40x (404 in the log message) which basically terminates the thread to<br/> keep retrying to make the registration.</p> <p>I'm not sure that retrying would help anyway. Something seems<br/> busted at this point in time, but where? This is after we have<br/> finished the ha_l3 robot suite which takes all three ODLs down at the<br/> same time and brings them back. The cluster is back and in sync ~12m<br/> before this happens, but in those 12m our haproxy never gets the websocket<br/> backup as UP. It does get the restconf backend up. You can see here <span class="error">[2]</span><br/> in the haproxy log where this happens. 20:21:29 shows the last of the <br/> 3 nodes going down for websocket, and after you only see restconf come<br/> back UP and no more messages about websocket.</p> <p>Now, in each of the karaf logs <span class="error">[3]</span><span class="error">[4]</span><span class="error">[5]</span> you can see that our haproxy<br/> healthcheck is hitting each node and it's rejecting it because the<br/> registration is not there yet. Remember, this is fluorine and your<br/> change to pre-register this websocket is not done. The point is that<br/> haproxy is there and working and polling on each node.</p> <p>So, where is that registration getting lost. The one from <span class="error">[1]</span>. Oh,<br/> here is the full neutron-server log to see it <span class="error">[6]</span>. It seems that the<br/> restconf registration request is being sent by networking-odl, but<br/> it's not working.</p> <p>That's as far as I got for now. Any ideas to run with?</p> <p>While I was writing this, one of my oxygen jobs <span class="error">[7]</span> also seems to have <br/> failed in the same way. So, my hope that websocket pre-registration<br/> changes that are there in oxygen would fix it, doesn't seem to.</p> <p>Here's a JIRA for this one:<br/> <a href="https://jira.opendaylight.org/browse/NEUTRON-204" class="external-link" rel="nofollow">https://jira.opendaylight.org/browse/NEUTRON-204</a></p> <p>Thanks,<br/> JamO</p> <p><span class="error">[0]</span> <a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/499/jamo-netvirt-csit-3node-0cmb-1ctl-2cmp-openstack-queens-upstream-stateful-fluorine/14" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/499/jamo-netvirt-csit-3node-0cmb-1ctl-2cmp-openstack-queens-upstream-stateful-fluorine/14</a><br/> <span class="error">[1]</span> <a href="http://git.openstack.org/cgit/openstack/networking-odl/tree/networking_odl/common/websocket_client.py?id=38497ef6c0c228c1794ddae3ad971353b3ff64c4#n115" class="external-link" target="_blank" rel="nofollow noopener">http://git.openstack.org/cgit/openstack/networking-odl/tree/networking_odl/common/websocket_client.py?id=38497ef6c0c228c1794ddae3ad971353b3ff64c4#n115</a><br/> <span class="error">[2]</span> <a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/499/jamo-netvirt-csit-3node-0cmb-1ctl-2cmp-openstack-queens-upstream-stateful-fluorine/14/haproxy.log.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/499/jamo-netvirt-csit-3node-0cmb-1ctl-2cmp-openstack-queens-upstream-stateful-fluorine/14/haproxy.log.gz</a><br/> <span class="error">[3]</span> <a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/499/jamo-netvirt-csit-3node-0cmb-1ctl-2cmp-openstack-queens-upstream-stateful-fluorine/14/odl_1/odl1_karaf.log.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/499/jamo-netvirt-csit-3node-0cmb-1ctl-2cmp-openstack-queens-upstream-stateful-fluorine/14/odl_1/odl1_karaf.log.gz</a><br/> <span class="error">[4]</span> <a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/499/jamo-netvirt-csit-3node-0cmb-1ctl-2cmp-openstack-queens-upstream-stateful-fluorine/14/odl_2/odl2_karaf.log.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/499/jamo-netvirt-csit-3node-0cmb-1ctl-2cmp-openstack-queens-upstream-stateful-fluorine/14/odl_2/odl2_karaf.log.gz</a><br/> <span class="error">[5]</span> <a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/499/jamo-netvirt-csit-3node-0cmb-1ctl-2cmp-openstack-queens-upstream-stateful-fluorine/14/odl_3/odl3_karaf.log.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/499/jamo-netvirt-csit-3node-0cmb-1ctl-2cmp-openstack-queens-upstream-stateful-fluorine/14/odl_3/odl3_karaf.log.gz</a><br/> <span class="error">[6]</span> <a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/499/jamo-netvirt-csit-3node-0cmb-1ctl-2cmp-openstack-queens-upstream-stateful-fluorine/14/control_1/oslogs/neutron-server.log.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/499/jamo-netvirt-csit-3node-0cmb-1ctl-2cmp-openstack-queens-upstream-stateful-fluorine/14/control_1/oslogs/neutron-server.log.gz</a><br/> <span class="error">[7]</span> <a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/500/jamo-netvirt-csit-3node-0cmb-1ctl-2cmp-openstack-queens-upstream-stateful-oxygen/12/" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/500/jamo-netvirt-csit-3node-0cmb-1ctl-2cmp-openstack-queens-upstream-stateful-oxygen/12/</a></p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=jhershbe" class="user-hover" rel="jhershbe">jhershbe</a>, did we ever get a patch in networking-odl to not actually die in the thread that<br/> retries the websocket? I think that's all we need to mark this specific bug as resolved.</p> Blocks GENIUS-263 Development Rank 0|i03k9j: