https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/NETVIRT-1460 netvirt <p>we have sporadic failures in our netvirt 3node (aka clustering) suites where openstack<br/> instances go to error state. the reason is shown as "Failed to allocate network(s)".</p> <p>In this example, this is happening after all three nodes have been stopped and started.<br/> The cluster is showing as up and operational, in that syncstatus is True and every shard<br/> has a proper Leader and Followers. Aprox 10m after the nodes are started these <br/> instances are booted that end up in error state.</p> <p>There is one ODL <a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-3node-0cmb-1ctl-2cmp-openstack-queens-upstream-stateful-snat-conntrack-oxygen/65/odl_3/odl3_karaf.log.gz" class="external-link" target="_blank" rel="nofollow noopener">karaf.log </a> with some of these messages:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> 2018-09-16T10:25:39,274 | ERROR | nioEventLoopGroup-7-1 | WebSocketServerHandler | 336 - org.opendaylight.netconf.restconf-nb-bierman02 - 1.7.4.SNAPSHOT | Listener <span class="code-keyword">for</span> stream with name <span class="code-quote">'data-change-event-subscription/neutron:neutron/neutron:ports/datastore=OPERATIONAL/scope=SUBTREE'</span> was not found. </pre> </div></div> <p>In the <a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-3node-0cmb-1ctl-2cmp-openstack-queens-upstream-stateful-snat-conntrack-oxygen/65/compute_2/oslogs/nova-compute.log.gz" class="external-link" target="_blank" rel="nofollow noopener">nova log </a> we can see some operation has timed out, so maybe that is because<br/> of the failing websocket:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> WARNING nova.virt.libvirt.driver [[01;36mNone req-a139053f-fabc-4c64-b658-0e73c1e4ecc5 [00;36madmin admin] [01;35m[instance: e115f123-25e9-4e6c-80be-347564d75af1] Timeout waiting <span class="code-keyword">for</span> [(<span class="code-quote">'network-vif-plugged'</span>, u<span class="code-quote">'8d8e28ca-1b96-41f9-8d12-6b041e5300e9'</span>)] <span class="code-keyword">for</span> instance with vm_state building and task_state spawning.[00m: Timeout: 300 seconds </pre> </div></div> <p>The other two ODL nodes do not seem to have this websocket error, but that may<br/> only be because just the one ODL is being hit with the requests (via haproxy). Or it<br/> may be that it's just not broken on the other two nodes.</p> NETVIRT-1460

websocket failing: causes instance creation failures

Bug High Confirmed Unresolved Jamo Luhrsen Jamo Luhrsen csit:3node Fri, 12 Oct 2018 23:06:37 +0000 Wed, 5 Dec 2018 01:03:41 +0000 Fluorine-SR2 Neon Oxygen-SR4 General 0 3 <p>This is almost certainly caused by haproxy. Connecting the websocket consists of two rest calls that go to the rest port (8081 in CSIT, IIRC) and a websocket connection that goes to port 8185. If the websocket lands on a different ODL node than the rest calls then the above error is emitted. </p> <p>Thanks for digging on this one Josh. I'm no expert in haproxy, but I think I can see how this scenario could<br/> happen for us based on our config. Here is the current config:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">global daemon group haproxy log /dev/log local0 maxconn 20480 pidfile /tmp/haproxy.pid ssl-<span class="code-keyword">default</span>-bind-ciphers !SSLv2:kEECDH:kRSA:kEDH:kPSK:+3DES:!aNULL:!eNULL:!MD5:!EXP:!RC4:!SEED:!IDEA:!DES ssl-<span class="code-keyword">default</span>-bind-options no-sslv3 no-tlsv10 stats socket /<span class="code-keyword">var</span>/lib/haproxy/stats mode 600 level user stats timeout 2m user haproxy defaults log global maxconn 4096 mode tcp retries 3 timeout http-request 10s timeout queue 2m timeout connect 10s timeout client 2m timeout server 2m timeout check 10s listen opendaylight bind 10.30.170.24:8181 transparent mode http http-request set-header X-Forwarded-Proto https <span class="code-keyword">if</span> { ssl_fc } http-request set-header X-Forwarded-Proto http <span class="code-keyword">if</span> !{ ssl_fc } option httpchk GET /diagstatus option httplog server opendaylight-<span class="code-keyword">rest</span>-1 10.30.170.101:8181 check fall 5 inter 2000 rise 2 server opendaylight-<span class="code-keyword">rest</span>-2 10.30.170.33:8181 check fall 5 inter 2000 rise 2 server opendaylight-<span class="code-keyword">rest</span>-3 10.30.170.134:8181 check fall 5 inter 2000 rise 2 listen opendaylight_ws bind 10.30.170.24:8185 transparent mode http timeout connect 5s timeout client 25s timeout server 25s timeout tunnel 3600s server opendaylight-ws-1 10.30.170.101:8185 check fall 5 inter 2000 rise 2 server opendaylight-ws-2 10.30.170.33:8185 check fall 5 inter 2000 rise 2 server opendaylight-ws-3 10.30.170.134:8185 check fall 5 inter 2000 rise 2 </pre> </div></div> <p>So, if Iunderstand it right, there are two different LBs happening here. one for<br/> 8181 and one for 8185, and the default algo is round robin. My guess is that<br/> there is no guarantee that both requests that are needed for a websocket<br/> create will hit the same server with the above config. And, maybe its possible<br/> (especially when downing nodes) that we get fully out of sync in the two<br/> round robin handling that we fail to create the websocket forever (or a really<br/> long time).</p> <p>For now, I'm <a href="https://git.opendaylight.org/gerrit/#/c/76955" class="external-link" target="_blank" rel="nofollow noopener">trying to tweak </a> the haproxy config to put both ports in the same<br/> listener and use source loadbalancing. This will not spread the load of across<br/> all the controllers, so maybe we have something else to try if that's the goal.<br/> But, if it works like we think, then all requests will end up going to the same<br/> ODL and websocket creation won't fail like we think it is.</p> <p>Also, I noticed that our existing config's listener section for 8185 does not<br/> have the /diagstatus healthcheck. So it might be possible that 8185 is<br/> still getting sent to a controller that is not really ready.</p> <p>Some more (and some repetitive) info from the mail thread...</p> <p> <br/> As Tim said, "IIRC the registration request just needs to hit every ODL, but after that neutron should be able to connect via websocket to any ODL." What this means is that the registration request and the websocket connection do not need to be handled by the same ODL instance. However, the ODL instance that handles the websocket connection <b>must</b> have received a registration request from some (any) networking_odl but not necessarily the one connecting the websocket. In a director deployment there are three neutron controllers and the same number of ODL instances. As such, given that the algorithm is RR there is at least a very high likelihood that a registration call will hit each ODL instance. In CSIT we have just one controller!<br/>  <br/> I think this is very wrong even for director despite the fact that we haven't noticed any issues yet. The reason we have HA is so that we can handle when some nodes croak. If a neutron controller node goes down, we will see this same phenomenon of websocket connections failing. IMHO, this needs to be fixed. Also, it feels like something that "works by accident." I see a few options:<br/>  <br/> 1) Configure haproxy so that rest and websocket connections from the same host will always get proxied to the same odl instance. Obviously, this should be done in such a way as to balance the traffic.<br/>  <br/> 2) Modify networking_odl so that the rest calls for registration are not sent via the VIP but rather explicitly sent to each odl node. The websocket connections would continue to use the VIP. At first, this seemed a bit ugly but on second thought I actually do not think this is so bad. The registration calls are not something that happen often or carry any kind of load. It is less than optimal that networking_odl would need to know about the real IPs but all in all this is a rather safe and straightforward fix. </p> <p><a href="https://git.opendaylight.org/gerrit/#/c/77229/" class="external-link" target="_blank" rel="nofollow noopener">this patch </a> seems to mask this issue</p> <p>will close this issue when <a href="https://git.opendaylight.org/gerrit/#/c/77569/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/77569/</a> is merged</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=shague" class="user-hover" rel="shague">shague</a>, can we keep this open? I know we had one patch merged towards fixing this general problem, but it's still not totally fixed.</p> <p><a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-3node-0cmb-1ctl-2cmp-openstack-queens-upstream-stateful-fluorine/135/robot-plugin/log_full.html.gz#s1-s6-t21-k3-k1-k3-k1-k2" class="external-link" target="_blank" rel="nofollow noopener">robot log showing instances in error state, failing to allocate network </a><br/> <a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-3node-0cmb-1ctl-2cmp-openstack-queens-upstream-stateful-fluorine/135/control_1/oslogs/neutron-server.log.gz" class="external-link" target="_blank" rel="nofollow noopener">neutron server log </a></p> <p>in the neutron log you can see the below message, which I think is indicative of this<br/> problem:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> 48506:2018-11-16 23:57:13.071 sERROR networking_odl.common.websocket_client None req-582e6941-f0b2-4e84-9be5-c8d7984be1cf None None websocket irrecoverable error </pre> </div></div> <p>yeah, that's fine to keep open. I closed it after I saw your comment about closing when 77569 is merged so I closed it.</p> Relates NETVIRT-1461 Development Rank 0|i03jlj: