[NETCONF-503] Listener for stream was not found Created: 24/Jan/18  Updated: 26/Jan/18  Resolved: 25/Jan/18

Status: Resolved
Project: netconf
Component/s: None
Affects Version/s: None
Fix Version/s: Carbon SR3

Type: Bug Priority: Medium
Reporter: Zhijiang Hu Assignee: Tim Rozet
Resolution: Won't Do Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

 The environment is a five-nodes OpenStack+ODL cluster which is deployed on a POD for OPNFV production CI(run every day, the latest failure is at http://artifacts.opnfv.org/logs/functest/zte-pod3/2018-01-23_21-02-30/functest.log).  Three nodes of them act as controllers for both ODL and OpenStack.  The other two nodes are act as compute nodes. All the five nodes have br-ex bridge that connect to physnet. The OVSs actively connect to each ODL controller.

I have collected all the three controller node's config files and logs at https://github.com/huzhijiang/odldebug0119  



 Description   

I am currently integrating OpenStack Pike with ODL Carbon( https://nexus.opendaylight.org/content/repositories/opendaylight-carbon-epel-7-x86_64-devel/org/opendaylight/integration-packaging/opendaylight/6.3.0-0.1.20180108snap438.el7.noarch/opendaylight-6.3.0-0.1.20180108snap438.el7.noarch.rpm).

When creating VM virtual interface, networking-odl through below error:

 

2018-01-18 11:42:14.174 33 DEBUG networking_odl.common.client [-] Sending METHOD (post) URL (http://192.168.11.20:8087/restconf/operations/sal-remote:create-data-change-event-subscription) JSON ({

  "input":

{     "path": "/neutron:neutron/neutron:ports",     "sal-remote-augment:notification-output-type": "JSON",     "sal-remote-augment:datastore": "OPERATIONAL",     "sal-remote-augment:scope": "SUBTREE"   }

}) request /usr/lib/python2.7/site-packages/networking_odl/common/client.py:89

2018-01-18 11:42:14.186 33 DEBUG networking_odl.common.client [-] Got response:

({"output":{"stream-name":"data-change-event-subscription/neutron:neutron/neutron:ports/datastore=OPERATIONAL/scope=SUBTREE"}}) _check_response /usr/lib/python2.7/site-packages/networking_odl/common/client.py:143

 

 

2018-01-18 11:42:14.187 33 DEBUG networking_odl.common.client [-] Sending METHOD (get) URL (http://192.168.11.20:8087/restconf/streams/stream/data-change-event-subscription/neutron:neutron/neutron:ports/datastore=OPERATIONAL/scope=SUBTREE?odl-leaf-nodes-only=true) JSON (None) request /usr/lib/python2.7/site-packages/networking_odl/common/client.py:89

2018-01-18 11:42:14.193 33 DEBUG networking_odl.common.websocket_client [-] websocket stream URL: ws://192.168.11.20:8185/data-change-event-subscription/neutron:neutron/neutron:ports/datastore=OPERATIONAL/scope=SUBTREE _subscribe_websocket /usr/lib/python2.7/site-packages/networking_odl/common/websocket_client.py:210

2018-01-18 11:42:14.196 33 ERROR networking_odl.common.websocket_client [-] webSocket bad status exception: WebSocketBadStatusException: Handshake status 500

2018-01-18 11:42:14.196 33 ERROR networking_odl.common.websocket_client Traceback (most recent call last):

2018-01-18 11:42:14.196 33 ERROR networking_odl.common.websocket_client   File "/usr/lib/python2.7/site-packages/networking_odl/common/websocket_client.py", line 238, in _socket_create_connection

2018-01-18 11:42:14.196 33 ERROR networking_odl.common.websocket_client     timeout=self.timeout)

2018-01-18 11:42:14.196 33 ERROR networking_odl.common.websocket_client   File "/usr/lib/python2.7/site-packages/websocket/_core.py", line 104, in create_connection

2018-01-18 11:42:14.196 33 ERROR networking_odl.common.websocket_client     websock.connect(url, **options)

2018-01-18 11:42:14.196 33 ERROR networking_odl.common.websocket_client   File "/usr/lib/python2.7/site-packages/websocket/_core.py", line 269, in connect

2018-01-18 11:42:14.196 33 ERROR networking_odl.common.websocket_client     self.handshake_response = handshake(self.sock, *addrs, **options)

2018-01-18 11:42:14.196 33 ERROR networking_odl.common.websocket_client   File "/usr/lib/python2.7/site-packages/websocket/_handshake.py", line 67, in handshake

2018-01-18 11:42:14.196 33 ERROR networking_odl.common.websocket_client     status, resp = _get_resp_headers(sock)

2018-01-18 11:42:14.196 33 ERROR networking_odl.common.websocket_client   File "/usr/lib/python2.7/site-packages/websocket/_handshake.py", line 123, in _get_resp_headers

2018-01-18 11:42:14.196 33 ERROR networking_odl.common.websocket_client     raise WebSocketBadStatusException("Handshake status %d", status)

2018-01-18 11:42:14.196 33 ERROR networking_odl.common.websocket_client WebSocketBadStatusException: Handshake status 500

2018-01-18 11:42:14.196 33 ERROR networking_odl.common.websocket_client

 

 

And there are a lot of  Listener for stream was not found error logs in ODL like below:

 

 

2018-01-18 03:51:44,998 | ERROR | tLoopGroup-11-70 | WebSocketServerHandler           | 280 - org.opendaylight.netconf.sal-rest-connector - 1.5.3.SNAPSHOT | Listener for stream with name 'data-change-event-subscription/neutron:neutron/neutron:ports/datastore=OPERATIONAL/scope=SUBTREE' was not found.

 

It is a intermittent issue.  The environment is a five-nodes OpenStack+ODL cluster which is deployed on a POD for OPNFV production CI(run every day, the latest failure is at http://artifacts.opnfv.org/logs/functest/zte-pod3/2018-01-23_21-02-30/functest.log).  Three nodes of them act as controllers for both ODL and OpenStack.  The other two nodes are act as compute nodes. All the five nodes have br-ex bridge that connect to physnet. The OVSs actively connect to each ODL controller.

I have collected all the three controller node's config files and logs at https://github.com/huzhijiang/odldebug0119  

 



 Comments   
Comment by Zhijiang Hu [ 25/Jan/18 ]

Got email from trozet and I got the idea, thus I need to setup haproxy as rest-connector's frontend and modify 10-rest-connector.xml  accordingly too. Many thanks to Tim , I will try it in OPNFV CI for several times and see the result.

 
 
 

Comment by Zhijiang Hu [ 25/Jan/18 ]

I proposed a patch to upstream  https://review.openstack.org/#/c/537805/ to address this configuration issue and patch and test it in OPNFV.

Comment by Jamo Luhrsen [ 25/Jan/18 ]

Can we double check if this is truly a carbon sr3 blocker?

Comment by Tim Rozet [ 25/Jan/18 ]

The issue was due to a configuration issue with ha proxy that is sitting in between Neutron and ODL.  The behavior from networking-odl and ODL is as expected.

Comment by Zhijiang Hu [ 26/Jan/18 ]

trozet with the configuration you suggested , the issue is gone. everything works fine now . Thank you!

Generated at Wed Feb 07 20:15:12 UTC 2024 using Jira 8.20.10#820010-sha1:ace47f9899e9ee25d7157d59aa17ab06aee30d3d.