[CONTROLLER-1357] Clustering: buyCar RPC is not working on shard followers. Created: 03/Jun/15  Updated: 06/Jun/15  Resolved: 06/Jun/15

Status: Resolved
Project: controller
Component/s: clustering
Affects Version/s: Lithium
Fix Version/s: None

Type: Bug
Reporter: Phillip Shea Assignee: Moiz Raja
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Operating System: All
Platform: All


External issue ID: 3566
Priority: High

 Description   

This was encountered in the clustering datastore suite.

The test was run on distribution-karaf-0.3.0-20150602.193518-2190.zip

The buyCar RPC fails when used on a Follower, but not a Leader.

Test suite: clustering datastore.
Test: 010_crud_on_leader.robot
Keyword: Purchase cars on Follower1

The results can be reproduced manually by populating the cars and people shards, then using the following post with Postman or DLUX. It will pass if localhost is a leader of all shards. It will fail if localhost is a follower of all shards. I'm not sure what will happen localhost is a leader for some shards, and a follower for others as it's not possible to force each combination of that configuration.

url:
http://localhost:8181/restconf/operations/car-purchase:buy-car
data:
{
"input" :

{ "car-purchase:person" : "/people:people/people:person[people:id='user11']", "car-purchase:person-id" : "user11", "car-purchase:car-id" : "11" }

}

Result:
<Response [500]>
{"errors":{"error":[

{"error-type":"application","error-tag":"operation-failed","error-message":"The operation encountered an unexpected error while executing."}

]}}

I saw this in the log. Not sure if it's relevant.

2015-06-02 13:04:41,345 | ERROR | lt-dispatcher-26 | Shard | 150 - org.opendaylight.controller.sal-akka-raft - 1.2.0.SNAPSHOT | member-3-shard-inventory-operational An exception occurred while preCommitting transaction member-1-txn-248
java.lang.IllegalArgumentException: Unmodified candidate should never be in the payload
at org.opendaylight.controller.cluster.datastore.DataTreeCandidatePayload.writeNode(DataTreeCandidatePayload.java:81)[157:org.opendaylight.controller.sal-distributed-datastore:1.2.0.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.DataTreeCandidatePayload.writeChildren(DataTreeCandidatePayload.java:60)[157:org.opendaylight.controller.sal-distributed-datastore:1.2.0.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.DataTreeCandidatePayload.writeNode(DataTreeCandidatePayload.java:74)[157:org.opendaylight.controller.sal-distributed-datastore:1.2.0.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.DataTreeCandidatePayload.writeChildren(DataTreeCandidatePayload.java:60)[157:org.opendaylight.controller.sal-distributed-datastore:1.2.0.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.DataTreeCandidatePayload.writeNode(DataTreeCandidatePayload.java:74)[157:org.opendaylight.controller.sal-distributed-datastore:1.2.0.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.DataTreeCandidatePayload.writeChildren(DataTreeCandidatePayload.java:60)[157:org.opendaylight.controller.sal-distributed-datastore:1.2.0.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.DataTreeCandidatePayload.writeNode(DataTreeCandidatePayload.java:74)[157:org.opendaylight.controller.sal-distributed-datastore:1.2.0.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.DataTreeCandidatePayload.writeChildren(DataTreeCandidatePayload.java:60)[157:org.opendaylight.controller.sal-distributed-datastore:1.2.0.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.DataTreeCandidatePayload.writeNode(DataTreeCandidatePayload.java:74)[157:org.opendaylight.controller.sal-distributed-datastore:1.2.0.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.DataTreeCandidatePayload.writeChildren(DataTreeCandidatePayload.java:60)[157:org.opendaylight.controller.sal-distributed-datastore:1.2.0.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.DataTreeCandidatePayload.writeNode(DataTreeCandidatePayload.java:74)[157:org.opendaylight.controller.sal-distributed-datastore:1.2.0.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.DataTreeCandidatePayload.writeChildren(DataTreeCandidatePayload.java:60)[157:org.opendaylight.controller.sal-distributed-datastore:1.2.0.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.DataTreeCandidatePayload.writeNode(DataTreeCandidatePayload.java:74)[157:org.opendaylight.controller.sal-distributed-datastore:1.2.0.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.DataTreeCandidatePayload.writeChildren(DataTreeCandidatePayload.java:60)[157:org.opendaylight.controller.sal-distributed-datastore:1.2.0.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.DataTreeCandidatePayload.create(DataTreeCandidatePayload.java:99)[157:org.opendaylight.controller.sal-distributed-datastore:1.2.0.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.Shard.continueCommit(Shard.java:319)[157:org.opendaylight.controller.sal-distributed-datastore:1.2.0.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator.doCommit(ShardCommitCoordinator.java:314)[157:org.opendaylight.controller.sal-distributed-datastore:1.2.0.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator.doCanCommit(ShardCommitCoordinator.java:271)[157:org.opendaylight.controller.sal-distributed-datastore:1.2.0.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator.handleCanCommit(ShardCommitCoordinator.java:229)[157:org.opendaylight.controller.sal-distributed-datastore:1.2.0.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator.handleForwardedReadyTransaction(ShardCommitCoordinator.java:138)[157:org.opendaylight.controller.sal-distributed-datastore:1.2.0.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.Shard.onReceiveCommand(Shard.java:229)[157:org.opendaylight.controller.sal-distributed-datastore:1.2.0.SNAPSHOT]
at akka.persistence.UntypedPersistentActor.onReceive(Eventsourced.scala:430)[147:com.typesafe.akka.persistence.experimental:2.3.10]
at org.opendaylight.controller.cluster.common.actor.MeteringBehavior.apply(MeteringBehavior.java:97)[149:org.opendaylight.controller.sal-clustering-commons:1.2.0.SNAPSHOT]
at akka.actor.ActorCell$$anonfun$become$1.applyOrElse(ActorCell.scala:534)[142:com.typesafe.akka.actor:2.3.10]
at akka.persistence.Recovery$State$class.process(Recovery.scala:30)[147:com.typesafe.akka.persistence.experimental:2.3.10]
at akka.persistence.ProcessorImpl$$anon$2.process(Processor.scala:103)[147:com.typesafe.akka.persistence.experimental:2.3.10]
at akka.persistence.ProcessorImpl$$anon$2.aroundReceive(Processor.scala:114)[147:com.typesafe.akka.persistence.experimental:2.3.10]
at akka.persistence.Recovery$class.aroundReceive(Recovery.scala:265)[147:com.typesafe.akka.persistence.experimental:2.3.10]
at akka.persistence.UntypedPersistentActor.akka$persistence$Eventsourced$$super$aroundReceive(Eventsourced.scala:428)[147:com.typesafe.akka.persistence.experimental:2.3.10]
at akka.persistence.Eventsourced$$anon$2.doAroundReceive(Eventsourced.scala:82)[147:com.typesafe.akka.persistence.experimental:2.3.10]
at akka.persistence.Eventsourced$$anon$2.aroundReceive(Eventsourced.scala:78)[147:com.typesafe.akka.persistence.experimental:2.3.10]
at akka.persistence.Eventsourced$class.aroundReceive(Eventsourced.scala:369)[147:com.typesafe.akka.persistence.experimental:2.3.10]
at akka.persistence.UntypedPersistentActor.aroundReceive(Eventsourced.scala:428)[147:com.typesafe.akka.persistence.experimental:2.3.10]
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)[142:com.typesafe.akka.actor:2.3.10]
at akka.actor.ActorCell.invoke(ActorCell.scala:487)[142:com.typesafe.akka.actor:2.3.10]
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:254)[142:com.typesafe.akka.actor:2.3.10]
at akka.dispatch.Mailbox.run(Mailbox.scala:221)[142:com.typesafe.akka.actor:2.3.10]
at akka.dispatch.Mailbox.exec(Mailbox.scala:231)[142:com.typesafe.akka.actor:2.3.10]
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)[139:org.scala-lang.scala-library:2.10.4.v20140209-180020-VFINAL-b66a39653b]
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.pollAndExecAll(ForkJoinPool.java:1253)[139:org.scala-lang.scala-library:2.10.4.v20140209-180020-VFINAL-b66a39653b]
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1346)[139:org.scala-lang.scala-library:2.10.4.v20140209-180020-VFINAL-b66a39653b]
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)[139:org.scala-lang.scala-library:2.10.4.v20140209-180020-VFINAL-b66a39653b]
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)[139:org.scala-lang.scala-library:2.10.4.v20140209-180020-VFINAL-b66a39653b]



 Comments   
Comment by Moiz Raja [ 04/Jun/15 ]

Looks like this commit broke remote-rpc https://git.opendaylight.org/gerrit/#/c/16754/3

Comment by Moiz Raja [ 06/Jun/15 ]

Fixes

https://git.opendaylight.org/gerrit/#/c/21890/
https://git.opendaylight.org/gerrit/#/c/22006/

Generated at Wed Feb 07 19:55:20 UTC 2024 using Jira 8.20.10#820010-sha1:ace47f9899e9ee25d7157d59aa17ab06aee30d3d.