Not able to perform RPC on a controller other than the one on which the shard was initialized.

Configuration:
3-node cluster
All features installed

This was first noticed in CI (both master and Lithium): https://jenkins.opendaylight.org/controller/job/controller-csit-3node-cds-clustering-all-stable-lithium/

The test is trying to add people using RPC and failing with a 500 error.

The difference between the default org.apache.karaf.features.cfg and the one used in the "all" configuration is as follows:

Default:
#

1. Comma separated list of features repositories to register by default
   #
   featuresRepositories = mvn:org.apache.karaf.features/standard/3.0.3/xml/features,mvn:org.apache.karaf.features/enterprise/3.0.3/xml/features,mvn:org.ops4j.pax.web/pax-web-features/3.1.4/xml/features,mvn:org.apache.karaf.features/spring/3.0.3/xml/features,mvn:org.opendaylight.integration/features-integration-index/ODL_DISTRIBUTION/xml/features
   #
2. Comma separated list of features to install at startup
   #
   featuresBoot=config,standard,region,package,kar,ssh,management,odl-restconf,odl-clustering-test-app,odl-netconf-connector-ssh

All:
#

1. Comma separated list of features repositories to register by default
   #
   featuresRepositories = mvn:org.apache.karaf.features/standard/3.0.3/xml/features,mvn:org.apache.karaf.features/enterprise/3.0.3/xml/features,mvn:org.ops4j.pax.web/pax-web-features/3.1.4/xml/features,mvn:org.apache.karaf.features/spring/3.0.3/xml/features,mvn:org.opendaylight.integration/features-integration-index/ODL_DISTRIBUTION/xml/features,mvn:org.opendaylight.integration/features-integration-test/ODL_DISTRIBUTION/xml/features
   #
2. Comma separated list of features to install at startup
   #
   featuresBoot=config,standard,region,package,kar,ssh,management,odl-integration-compatible-with-all,odl-restconf,odl-clustering-test-app,odl-netconf-connector-ssh

I was able to reproduce this locally by using the following two POST commands to intialize the shard, then populate it.

http://Controller1:8181/restconf/config
POST
{"people:people":{
"person": [

{ "id": "user0", "gender": "unknown", "age": "0", "address": "0Way, Some Country, Some Zip 0", "contactNo":"some number0" }

]
}}

http://controller2:8181/restconf/operations/people:add-person
POST
{
"input":

{ "people:id":"user999", "people:gender":"Male", "people:address":"San Jose", "people:contactNo":"555-1212", "people:age":"23" }

}

I've attached one of the logs. I'll find a place for the rest.
This message looks interesting:

java.lang.IllegalStateException: Conflicting instruction queue found

as does this one:

java.lang.IllegalStateException: Transaction chain has failed

and this one:

2015-08-07 15:33:04,584 | ERROR | CommitFutures-4 | NetconfDeviceTopologyAdapter | 223 - org.opendaylight.controller.sal-netconf-connector - 1.2.0.Lithium | RemoteDevice

{controller-config}

: TransactionChain(org.opendaylight.controller.md.sal.binding.impl.BindingDOMTransactionChainAdapter@3c9eb5c5) DOM-CHAIN-17-0 FAILED!
OptimisticLockFailedException

{message=Optimistic lock failed., errorList=[RpcError [message=Optimistic lock failed., severity=ERROR, errorType=APPLICATION, tag=resource-denied, applicationTag=null, info=null, cause=org.opendaylight.yangtools.yang.data.api.schema.tree.ConflictingModificationAppliedException: Node was deleted by other transaction.]]}