[CONTROLLER-1418] rpc buy-car failed with http status 501 Created: 28/Aug/15  Updated: 19/Oct/17  Resolved: 24/Feb/16

Status: Resolved
Project: controller
Component/s: clustering
Affects Version/s: Beryllium
Fix Version/s: None

Type: Bug
Reporter: Peter Gubka Assignee: Unassigned
Resolution: Cannot Reproduce Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Operating System: Linux
Platform: PC

Attachments: Text File karaf.log     File karaf_20160223.log.tar.xz    
External issue ID: 4220
Priority: Normal

 Description   

odl: http://nexus.opendaylight.org/content/repositories/opendaylight.snapshot/org/opendaylight/integration/distribution-karaf/0.3.2-SNAPSHOT//distribution-karaf-0.3.2-20150826.090317-16.tar.gz

steps to reproduce (taken from tcpdump:

1) ADDING PERSON
POST /restconf/config/people:people HTTP/1.1
Host: g32:8181
Accept-Encoding: identity
Content-Length: 95
Content-Type: application/json
Authorization: Basic YWRtaW46YWRtaW4=
{"person": [

{"contactNo": "1", "gender": "male", "age": "99", "id": 1, "address": "address1"}

]}

HTTP/1.1 204 No Content
Location: http://g32:8181/restconf/config/people:people
Server: Jetty(8.1.15.v20140411)

2) ADDING CAR
POST /restconf/config/car:cars HTTP/1.1
Host: g32:8181
Accept-Encoding: identity
Content-Length: 123
Content-Type: application/json
Authorization: Basic YWRtaW46YWRtaW4=
{"car-entry": [

{"category": "my_category", "model": "model1", "manufacturer": "my_manufacturer", "id": 1, "year": "2015"}

]}

HTTP/1.1 204 No Content
Location: http://g32:8181/restconf/config/car:cars
Server: Jetty(8.1.15.v20140411)

3) PURCHASE
POST /restconf/operations/car-purchase:buy-car HTTP/1.1
Host: g32:8181
Accept-Encoding: identity
Content-Length: 140
Content-Type: application/json
Authorization: Basic YWRtaW46YWRtaW4=

{"input": {"car-purchase:person": "/people:people/people:person[people:id='1']", "car-purchase:car-id": "1", "car-purchase:person-id": "1"}}

HTTP/1.1 500 Server Error
Content-Type: application/json
Transfer-Encoding: chunked
Server: Jetty(8.1.15.v20140411)

58F

{"errors":{"error":[{"error-type":"application","error-tag":"operation-failed","error-message":"The operation encountered an unexpected error while executing.","error-info":"org.opendaylight.controller.md.sal.dom.api.DOMRpcImplementationNotAvailableException: No local or remote implementation available for rpc AbsoluteSchemaPath

{path=[(urn:opendaylight:params:xml:ns:yang:controller:config:sal-clustering-it:car-purchase?revision=2014-08-18)buy-car]}

\n\tat org.opendaylight.controller.remote.rpc.RemoteRpcImplementation$1.onComplete(RemoteRpcImplementation.java:57)\n\tat org.opendaylight.controller.remote.rpc.RemoteRpcImplementation$1.onComplete(RemoteRpcImplementation.java:48)\n\tat akka.dispatch.OnComplete.internal(Future.scala:248)\n\tat akka.dispatch.OnComplete.internal(Future.scala:245)\n\tat akka.dispatch.japi$CallbackBridge.apply(Future.scala:175)\n\tat akka.dispatch.japi$CallbackBridge.apply(Future.scala:172)\n\tat scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)\n\tat scala.concurrent.impl.ExecutionContextImpl$$anon$3.exec(ExecutionContextImpl.scala:107)\n\tat scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)\n\tat scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)\n\tat scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)\n\tat scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)\n"}]}}

0

Get cars and persons returns
<cars xmlns="urn:opendaylight:params:xml:ns:yang:controller:config:sal-clustering-it:car"><car-entry><id>1</id><year>2015</year><category>my_category</category><manufacturer>my_manufacturer</manufacturer><model>model1</model></car-entry></cars>
Out[19]: <Response [200]>

<people xmlns="urn:opendaylight:params:xml:ns:yang:controller:config:sal-clustering-it:people"><person><id>1</id><contactNo>1</contactNo><address>address1</address><age>99</age><gender>male</gender></person></people>

But buying a car with non existing user id "user1" pass, but buying with correct id "1" fails with 500 (see above)

POST /restconf/operations/car-purchase:buy-car HTTP/1.1
Host: g32:8181
Content-Length: 230
Accept-Encoding: gzip, deflate
Accept: /
User-Agent: python-requests/2.4.1 CPython/2.7.6 Linux/3.13.0-55-generic
Connection: keep-alive
Content-Type: application/json
Authorization: Basic YWRtaW46YWRtaW4=
{
"input" :

{ "car-purchase:person" : "/people:people/people:person[people:id='user1']", "car-purchase:person-id" : "user1", "car-purchase:car-id" : "1" }

}

HTTP/1.1 200 OK
Content-Type: application/yang.operation+json
Content-Length: 0
Server: Jetty(8.1.15.v20140411)

 Comments   
Comment by Peter Gubka [ 28/Aug/15 ]

Attachment karaf.log has been added with description: karaf log

Comment by Robert Varga [ 13/Nov/15 ]

Move to NETCONF project

Comment by Vratko Polak [ 11/Feb/16 ]

According to [0], "no RPC implementation" is expected to happen just after ODL starts, and users should keep retrying until they get different response.

Also, on Beryllium the HTTP status code is 501, which sounds about right.

Setting to FIXED, may be re-opened if tests do not confirm the operation finds an implementation in reasonable time.

[0] https://lists.opendaylight.org/pipermail/integration-dev/2016-February/005824.html

Comment by Vratko Polak [ 11/Feb/16 ]

Re-opening, as in 1-node setup (default clustering configs, e.g. without separate car, people and car-people shards) even 20 minutes is not enough for 501 to go away.
A suite for checking that is being developed, here is a link to how it fared on Sandbox: [0]. Non-RPC operations (adding and removing cars and people to/from config datastore) work well.

The 3-node test job shows this is a fairly recent regression [1], two days ago the test was passing.

Updated Bug title. Perhaps the new breakage is in controller project?

[0] https://jenkins.opendaylight.org/sandbox/job/integration-csit-verify-1node-library/25/robot/report/log.html#s1-s2-t1-k2-k61-k1-k1-k1-k7-k1
[1] https://jenkins.opendaylight.org/releng/view/controller/job/controller-csit-3node-clustering-only-beryllium/571/robot/controller-clustering.txt/010%20Crud%20On%20Leader/Purchase%20Cars%20On%20Leader/

Comment by Vratko Polak [ 12/Feb/16 ]

3-node suite started passing again, 1-node suite still fails.

Comment by Vratko Polak [ 12/Feb/16 ]

Keywords for testcase-to-bug search:
carpeople_library_test_wait_for_rpcs
010_crud_on_leader_purchase_cars_on_leader

The 1-node suite: https://git.opendaylight.org/gerrit/33987

Comment by Vratko Polak [ 18/Feb/16 ]

The 1-node suite fails also with 34175 [0] applied.

Additional keyword: test_libraries_txt_carpeople_library_test_wait_for_rpcs

[0] https://git.opendaylight.org/gerrit/34175

Comment by Vratko Polak [ 24/Feb/16 ]

I have tested this some more. Beryllium snapshot build from yesterday, 1-node, every config (except logging) default, bootFeatures: odl-restconf,odl-clustering-test-app,odl-jolokia

karaf.log shows buy-car RPC was registered (several times in fact) during ODL boot, but request still gets DOMRpcImplementationNotAvailableException.

Setting as controller/clustering bug. There was a WARN 5 minutes after request (repeated request also failed), so there may be something wrong in Restconf too, but the exception comes from org.opendaylight.controller.remote.rpc.

Comment by Vratko Polak [ 24/Feb/16 ]

Attachment karaf_20160223.log.tar.xz has been added with description: Archive with karaf.log Comment 7 is referring to

Comment by Tom Pantelis [ 24/Feb/16 ]

What is .xz extension - how do I decrypt that? Or maybe you can attach the normal karaf.log?

(In reply to Vratko Polák from comment #8)
> Created attachment 849 [details]
> Archive with karaf.log Comment 7 is referring to

Comment by Tom Pantelis [ 24/Feb/16 ]

Is all your testing done with the automated tests? If so can you reproduce manually? I'm trying to follow all of this but I'm not clear on how to reproduce exactly.

Comment by Tom Pantelis [ 24/Feb/16 ]

I followed the reproduction steps in the bug description and got the 501. The reason there's no routed RPC registration is b/c it gets registered via the add-person RPC. So instead of POSTing to people:people, you need to invoke add-person, eg

restconf/operations/people:add-person
{
"input":

{ "people:id" : "1" }

}

Then buy-car succeeds. You'll notice the integration tests do this.

Comment by Vratko Polak [ 24/Feb/16 ]

> What is .xz extension

https://en.wikipedia.org/wiki/Xz

> how do I decrypt that?

On Linux, "tar" utility recognizes it, so "tar xf name.tar.xz" works.

> Is all your testing done with the automated tests?

Up to Comment 6 yes. Comment 7 was a manual test. After editing bootFeatures and logging, I did bin/start then I monitored karaf.log till it stopped scrolling and then this curl in bash (so internal single quotes look funny):

curl -v -u 'admin:admin' -X POST -H "Content-Type:application/yang.data+json" -d '{"input":{"person":"/people:people/people:person[people:id='"'"'joe'"'"']","car-id":"boogie","person-id":"joe"}}' 127.0.0.1:8181/restconf/operations/car-purchase:buy-car

The "joe" person was not created beforehand, but I think that should not "unregister" the RPC.
When I tried smaller payloads, restconf replied with 400 telling me why my data does not match the model.

Comment by Vratko Polak [ 24/Feb/16 ]

> it gets registered via the add-person RPC

Yup, that is what I missed.
Is there a wikipage which mentions this restriction?

Comment by Tom Pantelis [ 24/Feb/16 ]

No wiki page that I know of. The cluster-test-app is not intended for general use - it was developed specifically for the robot tests. Routed RPCs are not registered automatically by md-sal - there has to be app code to do it so that's the purpose of the add-person RPC. Also the context-reference for the routed buy-car RPC refers to a person so the person path must point to a valid person node.

(In reply to Vratko Polák from comment #13)
> > it gets registered via the add-person RPC
>
> Yup, that is what I missed.
> Is there a wikipage which mentions this restriction?

Generated at Wed Feb 07 19:55:30 UTC 2024 using Jira 8.20.10#820010-sha1:ace47f9899e9ee25d7157d59aa17ab06aee30d3d.