[CONTROLLER-1258] Clustering: RPC failing in clustering data store integration tests Created: 17/Apr/15  Updated: 05/May/15  Resolved: 05/May/15

Status: Resolved
Project: controller
Component/s: mdsal
Affects Version/s: Post-Helium
Fix Version/s: None

Type: Bug
Reporter: Phillip Shea Assignee: Harman Singh
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Operating System: All
Platform: All

External issue ID: 3016

 Description   

I'm running /suites/clustering/datastore/020_crud_on_leader.txt.

The Keywords that are failing are:

Purchase cars on Leader
Purchase cars on Follower1
Purchase cars on Follower2

What's happening is that the RPCs to "buy a car" only work when executed against one of the three controllers. Strangely it's note even the car-people shard leader.

Tom Pantelis is running these test in Brocade's lab on Helium and they are passing. This bug is unique to Lithium.

Procedure:

Pull the latest stable Lithium Release. I pulled distribution-karaf-0.3.0-20150414.220637-1140.zip

Use the python cluster deploy script.

Let the test run. The shards configured themselves in this way:

car shard

{'member1': 'Follower', 'member2': 'Leader', 'member3': 'Follower'}

people shard

{'member1': 'Follower', 'member2': 'Follower', 'member3': 'Leader'}

car-people shard

{'member1': 'Leader', 'member2': 'Follower', 'member3': 'Follower'}

After the test completes, use ReST RPCs to purchase a car on member1, member2, and member2.

URL:
http://member1:8181/restconf/operations/car-purchase:buy-car

JSON:
{ "input" :

{ "car-purchase:person" : "/people:people/people:person[people:id='user9']", "car-purchase:person-id" : "user9", "car-purchase:car-id" : "9" }

}

For the first two members, there will be a 500 error in Postman, and nothing in the karaf log (there will be a separate bug addressing the log and error messages). Member3 gets a nice “200 OK” in Postman and I see an acknowledgement of the car purchase in the karaf log.

RPCs are not working on two out of the three controllers.

 Comments   
Comment by Harman Singh [ 30/Apr/15 ]

Tony, I investigated this issue before and to fix it, you have to change serialization in
yang/yang-data-api/src/main/java/org/opendaylight/yangtools/yang/data/api/YangInstanceIdentifier.java

This serialization works fine in unit tests, but with routed rpc,
remote server was not able to deserialize the object in AKKA messages properly. I have to switch to manual serialization of immutablemap.

I made following changes, then it was working fine,

1. make ImmutableList<PathArgument> legacyPath volatile

private transient volatile ImmutableList<PathArgument> legacyPath = null;

2. Change readObject, add following line -

legacyPath = ImmutableList.copyOf((Collection<PathArgument>)inputStream.readObject());

3. Change writeObject to following -

ImmutableList<PathArgument> pathArguments = getLegacyPath(); outputStream.defaultWriteObject();
outputStream.writeObject(pathArguments);

Comment by Tony Tkacik [ 30/Apr/15 ]

Harman could you provide patch to yangtools?

Comment by Moiz Raja [ 30/Apr/15 ]

If I recall right this happened due to some Class Loader issue where the code that was deserializing did not have access to Guava or something. This may be tricky to reproduce in a simple unit test.

Comment by Harman Singh [ 30/Apr/15 ]

Here is the patch -

https://git.opendaylight.org/gerrit/#/c/19415/

Yes Moiz, this is the same issue and we see following exception -
ERROR Remoting - cannot assign instance of com.google.common.collect.RegularImmutableList
to field org.opendaylight.yangtools.yang.data.api.YangInstanceIdentifier.legacyPath of type
com.google.common.collect.ImmutableList in instance of org.opendaylight.yangtools.yang.data.api.YangInstanceIdentifier

Generated at Wed Feb 07 19:55:04 UTC 2024 using Jira 8.20.10#820010-sha1:ace47f9899e9ee25d7157d59aa17ab06aee30d3d.