This is almost certainly caused by haproxy. Connecting the websocket consists of two rest calls that go to the rest port (8081 in CSIT, IIRC) and a websocket connection that goes to port 8185. If the websocket lands on a different ODL node than the rest calls then the above error is emitted. 

Thanks for digging on this one Josh. I'm no expert in haproxy, but I think I can see how this scenario could
happen for us based on our config. Here is the current config:

global
  daemon
  group  haproxy
  log  /dev/log local0
  maxconn  20480
  pidfile  /tmp/haproxy.pid
  ssl-default-bind-ciphers  !SSLv2:kEECDH:kRSA:kEDH:kPSK:+3DES:!aNULL:!eNULL:!MD5:!EXP:!RC4:!SEED:!IDEA:!DES
  ssl-default-bind-options  no-sslv3 no-tlsv10
  stats  socket /var/lib/haproxy/stats mode 600 level user
  stats  timeout 2m
  user  haproxy

defaults
  log  global
  maxconn  4096
  mode  tcp
  retries  3
  timeout  http-request 10s
  timeout  queue 2m
  timeout  connect 10s
  timeout  client 2m
  timeout  server 2m
  timeout  check 10s

listen opendaylight
  bind 10.30.170.24:8181 transparent
  mode http
  http-request set-header X-Forwarded-Proto https if { ssl_fc }
  http-request set-header X-Forwarded-Proto http if !{ ssl_fc }
  option httpchk GET /diagstatus
  option httplog
  server opendaylight-rest-1 10.30.170.101:8181 check fall 5 inter 2000 rise 2
  server opendaylight-rest-2 10.30.170.33:8181 check fall 5 inter 2000 rise 2
  server opendaylight-rest-3 10.30.170.134:8181 check fall 5 inter 2000 rise 2

listen opendaylight_ws
  bind 10.30.170.24:8185 transparent
  mode http
  timeout connect 5s
  timeout client 25s
  timeout server 25s
  timeout tunnel 3600s
  server opendaylight-ws-1 10.30.170.101:8185 check fall 5 inter 2000 rise 2
  server opendaylight-ws-2 10.30.170.33:8185 check fall 5 inter 2000 rise 2
  server opendaylight-ws-3 10.30.170.134:8185 check fall 5 inter 2000 rise 2

So, if Iunderstand it right, there are two different LBs happening here. one for
8181 and one for 8185, and the default algo is round robin. My guess is that
there is no guarantee that both requests that are needed for a websocket
create will hit the same server with the above config. And, maybe its possible
(especially when downing nodes) that we get fully out of sync in the two
round robin handling that we fail to create the websocket forever (or a really
long time).

For now, I'm trying to tweak the haproxy config to put both ports in the same
listener and use source loadbalancing. This will not spread the load of across
all the controllers, so maybe we have something else to try if that's the goal.
But, if it works like we think, then all requests will end up going to the same
ODL and websocket creation won't fail like we think it is.

Also, I noticed that our existing config's listener section for 8185 does not
have the /diagstatus healthcheck. So it might be possible that 8185 is
still getting sent to a controller that is not really ready.

Some more (and some repetitive) info from the mail thread...

 
As Tim said, "IIRC the registration request just needs to hit every ODL, but after that neutron should be able to connect via websocket to any ODL." What this means is that the registration request and the websocket connection do not need to be handled by the same ODL instance. However, the ODL instance that handles the websocket connection must have received a registration request from some (any) networking_odl but not necessarily the one connecting the websocket. In a director deployment there are three neutron controllers and the same number of ODL instances. As such, given that the algorithm is RR there is at least a very high likelihood that a registration call will hit each ODL instance. In CSIT we have just one controller!
 
I think this is very wrong even for director despite the fact that we haven't noticed any issues yet. The reason we have HA is so that we can handle when some nodes croak. If a neutron controller node goes down, we will see this same phenomenon of websocket connections failing. IMHO, this needs to be fixed. Also, it feels like something that "works by accident." I see a few options:
 
1) Configure haproxy so that rest and websocket connections from the same host will always get proxied to the same odl instance. Obviously, this should be done in such a way as to balance the traffic.
 
2) Modify networking_odl so that the rest calls for registration are not sent via the VIP but rather explicitly sent to each odl node. The websocket connections would continue to use the VIP. At first, this seemed a bit ugly but on second thought I actually do not think this is so bad. The registration calls are not something that happen often or carry any kind of load. It is less than optimal that networking_odl would need to know about the real IPs but all in all this is a rather safe and straightforward fix. 

this patch seems to mask this issue

will close this issue when https://git.opendaylight.org/gerrit/#/c/77569/ is merged

shague, can we keep this open? I know we had one patch merged towards fixing this general problem, but it's still not totally fixed.

robot log showing instances in error state, failing to allocate network
neutron server log

in the neutron log you can see the below message, which I think is indicative of this
problem:

48506:2018-11-16 23:57:13.071 sERROR networking_odl.common.websocket_client None req-582e6941-f0b2-4e84-9be5-c8d7984be1cf None None websocket irrecoverable error 

yeah, that's fine to keep open. I closed it after I saw your comment about closing when 77569 is merged so I closed it.