This is related to CONTROLLER-1258. In addition to the problem encountered in the bug, it lacked a useful error message and log entry. This bug should be fixed first, since the other bug is needed to reproduce it.

I'm encountered this running /suites/clustering/datastore/020_crud_on_leader.txt.

The Keywords that are failing are:

Purchase cars on Leader
Purchase cars on Follower1
Purchase cars on Follower2

What's happening is that the RPCs to "buy a car" only work when executed against one of the three controllers. Strangely it's not even the car-people shard leader.

Tom Pantelis is running these test in Brocade's lab on Helium and they are passing. This bug is unique to Lithium.

Other than the fact that these keywords are failing, this bug addresses the issue that the error messages and log entries are missing when the "buy a car" RPCs are used.

Procedure:

Pull the latest stable Lithium Release. I pulled distribution-karaf-0.3.0-20150414.220637-1140.zip

Use the python cluster deploy script.

Let the test run. The shards configured themselves in this way:

car shard

{'member1': 'Follower', 'member2': 'Leader', 'member3': 'Follower'}

people shard

{'member1': 'Follower', 'member2': 'Follower', 'member3': 'Leader'}

car-people shard

{'member1': 'Leader', 'member2': 'Follower', 'member3': 'Follower'}

After the test completes, use ReST RPCs to purchase a car on member1, member2, and member2.

URL:
http://member1:8181/restconf/operations/car-purchase:buy-car

JSON:
{ "input" :

{ "car-purchase:person" : "/people:people/people:person[people:id='user9']", "car-purchase:person-id" : "user9", "car-purchase:car-id" : "9" }

}

For the first two members, there will be a 500 error in Postman, and nothing in the karaf log. Member3 gets a nice “200 OK” in Postman and I see an acknowledgement of the car purchase in the karaf log.

The 500 error should contain some text (or even a stack trace - if that's not a security issue) describing the reason for the error. In this case the 500 error has no text.

To observe the log files connect to the terminal on all three controllers.

ssh karaf@member1 -p 8101
ssh karaf@member2 -p 8101
ssh karaf@member3 -p 8101

If you try the "buy a car" RPCs, you will see nothing from the failures on member1 and member2, but you will see a acknowledgement message for the successful RPC on member3.

The failures should produce both errors in the RPC ReST call and in the karaf log.