[NETCONF-503] Listener for stream was not found Created: 24/Jan/18 Updated: 26/Jan/18 Resolved: 25/Jan/18 |
|
| Status: | Resolved |
| Project: | netconf |
| Component/s: | None |
| Affects Version/s: | None |
| Fix Version/s: | Carbon SR3 |
| Type: | Bug | Priority: | Medium |
| Reporter: | Zhijiang Hu | Assignee: | Tim Rozet |
| Resolution: | Won't Do | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Environment: |
The environment is a five-nodes OpenStack+ODL cluster which is deployed on a POD for OPNFV production CI(run every day, the latest failure is at http://artifacts.opnfv.org/logs/functest/zte-pod3/2018-01-23_21-02-30/functest.log). Three nodes of them act as controllers for both ODL and OpenStack. The other two nodes are act as compute nodes. All the five nodes have br-ex bridge that connect to physnet. The OVSs actively connect to each ODL controller. I have collected all the three controller node's config files and logs at https://github.com/huzhijiang/odldebug0119 |
||
| Description |
|
I am currently integrating OpenStack Pike with ODL Carbon( https://nexus.opendaylight.org/content/repositories/opendaylight-carbon-epel-7-x86_64-devel/org/opendaylight/integration-packaging/opendaylight/6.3.0-0.1.20180108snap438.el7.noarch/opendaylight-6.3.0-0.1.20180108snap438.el7.noarch.rpm). When creating VM virtual interface, networking-odl through below error:
2018-01-18 11:42:14.174 33 DEBUG networking_odl.common.client [-] Sending METHOD (post) URL (http://192.168.11.20:8087/restconf/operations/sal-remote:create-data-change-event-subscription) JSON ({ "input": { "path": "/neutron:neutron/neutron:ports", "sal-remote-augment:notification-output-type": "JSON", "sal-remote-augment:datastore": "OPERATIONAL", "sal-remote-augment:scope": "SUBTREE" }}) request /usr/lib/python2.7/site-packages/networking_odl/common/client.py:89 2018-01-18 11:42:14.186 33 DEBUG networking_odl.common.client [-] Got response: ({"output":{"stream-name":"data-change-event-subscription/neutron:neutron/neutron:ports/datastore=OPERATIONAL/scope=SUBTREE"}}) _check_response /usr/lib/python2.7/site-packages/networking_odl/common/client.py:143
2018-01-18 11:42:14.187 33 DEBUG networking_odl.common.client [-] Sending METHOD (get) URL (http://192.168.11.20:8087/restconf/streams/stream/data-change-event-subscription/neutron:neutron/neutron:ports/datastore=OPERATIONAL/scope=SUBTREE?odl-leaf-nodes-only=true) JSON (None) request /usr/lib/python2.7/site-packages/networking_odl/common/client.py:89 2018-01-18 11:42:14.193 33 DEBUG networking_odl.common.websocket_client [-] websocket stream URL: ws://192.168.11.20:8185/data-change-event-subscription/neutron:neutron/neutron:ports/datastore=OPERATIONAL/scope=SUBTREE _subscribe_websocket /usr/lib/python2.7/site-packages/networking_odl/common/websocket_client.py:210 2018-01-18 11:42:14.196 33 ERROR networking_odl.common.websocket_client [-] webSocket bad status exception: WebSocketBadStatusException: Handshake status 500 2018-01-18 11:42:14.196 33 ERROR networking_odl.common.websocket_client Traceback (most recent call last): 2018-01-18 11:42:14.196 33 ERROR networking_odl.common.websocket_client File "/usr/lib/python2.7/site-packages/networking_odl/common/websocket_client.py", line 238, in _socket_create_connection 2018-01-18 11:42:14.196 33 ERROR networking_odl.common.websocket_client timeout=self.timeout) 2018-01-18 11:42:14.196 33 ERROR networking_odl.common.websocket_client File "/usr/lib/python2.7/site-packages/websocket/_core.py", line 104, in create_connection 2018-01-18 11:42:14.196 33 ERROR networking_odl.common.websocket_client websock.connect(url, **options) 2018-01-18 11:42:14.196 33 ERROR networking_odl.common.websocket_client File "/usr/lib/python2.7/site-packages/websocket/_core.py", line 269, in connect 2018-01-18 11:42:14.196 33 ERROR networking_odl.common.websocket_client self.handshake_response = handshake(self.sock, *addrs, **options) 2018-01-18 11:42:14.196 33 ERROR networking_odl.common.websocket_client File "/usr/lib/python2.7/site-packages/websocket/_handshake.py", line 67, in handshake 2018-01-18 11:42:14.196 33 ERROR networking_odl.common.websocket_client status, resp = _get_resp_headers(sock) 2018-01-18 11:42:14.196 33 ERROR networking_odl.common.websocket_client File "/usr/lib/python2.7/site-packages/websocket/_handshake.py", line 123, in _get_resp_headers 2018-01-18 11:42:14.196 33 ERROR networking_odl.common.websocket_client raise WebSocketBadStatusException("Handshake status %d", status) 2018-01-18 11:42:14.196 33 ERROR networking_odl.common.websocket_client WebSocketBadStatusException: Handshake status 500 2018-01-18 11:42:14.196 33 ERROR networking_odl.common.websocket_client
And there are a lot of Listener for stream was not found error logs in ODL like below:
2018-01-18 03:51:44,998 | ERROR | tLoopGroup-11-70 | WebSocketServerHandler | 280 - org.opendaylight.netconf.sal-rest-connector - 1.5.3.SNAPSHOT | Listener for stream with name 'data-change-event-subscription/neutron:neutron/neutron:ports/datastore=OPERATIONAL/scope=SUBTREE' was not found.
It is a intermittent issue. The environment is a five-nodes OpenStack+ODL cluster which is deployed on a POD for OPNFV production CI(run every day, the latest failure is at http://artifacts.opnfv.org/logs/functest/zte-pod3/2018-01-23_21-02-30/functest.log). Three nodes of them act as controllers for both ODL and OpenStack. The other two nodes are act as compute nodes. All the five nodes have br-ex bridge that connect to physnet. The OVSs actively connect to each ODL controller. I have collected all the three controller node's config files and logs at https://github.com/huzhijiang/odldebug0119
|
| Comments |
| Comment by Zhijiang Hu [ 25/Jan/18 ] |
|
Got email from trozet and I got the idea, thus I need to setup haproxy as rest-connector's frontend and modify 10-rest-connector.xml accordingly too. Many thanks to Tim |
| Comment by Zhijiang Hu [ 25/Jan/18 ] |
|
I proposed a patch to upstream https://review.openstack.org/#/c/537805/ to address this configuration issue and patch and test it in OPNFV. |
| Comment by Jamo Luhrsen [ 25/Jan/18 ] |
|
Can we double check if this is truly a carbon sr3 blocker? |
| Comment by Tim Rozet [ 25/Jan/18 ] |
|
The issue was due to a configuration issue with ha proxy that is sitting in between Neutron and ODL. The behavior from networking-odl and ODL is as expected. |
| Comment by Zhijiang Hu [ 26/Jan/18 ] |
|
trozet with the configuration you suggested , the issue is gone. everything works fine now . Thank you! |