IllegalStateException [Address Not Found] is thrown for flow/group rpc operations.

Setup: 2 controllers C1 and C2 in cluster mode. C1 manages switch S1.

While trying to configure a flow from C2 to S1 [command: addMDFlow openflow:1 f1 1], the following exception is hit in C2:

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>.
2014-04-04 16:19:25.081 IST [md-sal-binding-commit-0] ERROR o.o.c.m.s.c.i.s.TwoPhaseCommit - Transaction: BA-113 Finish Commit failed
java.lang.IllegalStateException: Address not found for route [RouteIdentifierImpl{context=null, type=(urn:opendaylight:flow:service?revision=2013-08-19)add-flow, route=/(urn:opendaylight:inventory?revision=2013-08-19)nodes/(urn:opendaylight:inventory?revision=2013-08-19)node[

]}]
at com.google.common.base.Preconditions.checkState(Preconditions.java:176) ~[bundlefile:na]
at org.opendaylight.controller.sal.connector.remoterpc.ClientImpl.lookupRemoteAddressForRpc(ClientImpl.java:216) ~[na:na]
at org.opendaylight.controller.sal.connector.remoterpc.ClientImpl.invokeRpc(ClientImpl.java:124) ~[na:na]
at org.opendaylight.controller.sal.connector.remoterpc.RemoteRpcProvider.invokeRpc(RemoteRpcProvider.java:105) ~[na:na]
at org.opendaylight.controller.sal.dom.broker.impl.SchemaAwareRpcBroker.invokeRpc(SchemaAwareRpcBroker.java:240) ~[na:na]
at org.opendaylight.controller.sal.dom.broker.impl.SchemaAwareRpcBroker$RoutedRpcSelector.invokeRpc(SchemaAwareRpcBroker.java:337) ~[na:na]
at org.opendaylight.controller.sal.dom.broker.impl.SchemaAwareRpcBroker.invokeRpc(SchemaAwareRpcBroker.java:167) ~[na:na]
at org.opendaylight.controller.sal.dom.broker.osgi.RpcProvisionRegistryProxy.invokeRpc(RpcProvisionRegistryProxy.java:62) ~[na:na]
at org.opendaylight.controller.sal.binding.impl.connect.dom.BindingIndependentConnector$DefaultInvocationStrategy.forwardToDomBroker(BindingIndependentConnector.java:817) ~[na:na]
at org.opendaylight.controller.sal.binding.impl.connect.dom.BindingIndependentConnector$DomToBindingRpcForwarder.invoke(BindingIndependentConnector.java:666) ~[na:na]
at com.sun.proxy.$Proxy100.addFlow(Unknown Source) ~[na:na]
at org.opendaylight.yang.gen.v1.urn.opendaylight.flow.service.rev130819.SalFlowService$$Broker$Router.addFlow(SalFlowService$$Broker$Router.java) ~[na:na]
at org.opendaylight.yang.gen.v1.urn.opendaylight.flow.service.rev130819.SalFlowService$$Broker$DirectProxy.addFlow(SalFlowService$$Broker$DirectProxy.java) ~[na:na]
at org.opendaylight.controller.frm.flow.FlowTransaction.add(FlowTransaction.java:121) ~[na:na]
at org.opendaylight.controller.frm.AbstractTransaction.callRpcs(AbstractTransaction.java:63) ~[na:na]
at org.opendaylight.controller.frm.AbstractTransaction.finish(AbstractTransaction.java:35) ~[na:na]
at org.opendaylight.controller.md.sal.common.impl.service.TwoPhaseCommit.call(TwoPhaseCommit.java:110) [bundlefile:na]
at org.opendaylight.controller.md.sal.common.impl.service.TwoPhaseCommit.call(TwoPhaseCommit.java:38) [bundlefile:na]
at java.util.concurrent.FutureTask.run(FutureTask.java:262) [na:1.7.0_51]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_51]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_51]
at java.lang.Thread.run(Thread.java:744) [na:1.7.0_51]

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

Observations:
The issue is seen in 3 different scenarios:

1. this.supportedRpcs = mappingService.getRpcQNamesFor(service); in BindingIndependentConnector.java::DomToBindingRpcForwarder(Class<? extends RpcService> service, Class<? extends BaseIdentity> context) is returns an empty Set for RpcService value "SalFlowService".
a. We see entries for meter, table and stats for the specific switch in the rpcCache when we hit the Address not found problem. The only entries missing are flow and group.
b. In this scenario, if C2 manages switch S2 and C1 routes rpc from C1 to S2, following exception will be hit in C1:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2014-04-04 19:18:18.253 IST [md-sal-binding-commit-0] ERROR o.o.c.m.s.c.i.s.TwoPhaseCommit - Transaction: BA-113 Finish Commit failed
java.lang.IllegalStateException: null
at com.google.common.base.Preconditions.checkState(Preconditions.java:133) ~[bundlefile:na]
at org.opendaylight.controller.sal.binding.impl.connect.dom.BindingIndependentConnector$DomToBindingRpcForwarder.invoke(BindingIndependentConnector.java:694) ~[na:na]
at com.sun.proxy.$Proxy98.addFlow(Unknown Source) ~[na:na]
at org.opendaylight.yang.gen.v1.urn.opendaylight.flow.service.rev130819.SalFlowService$$Broker$Router.addFlow(SalFlowService$$Broker$Router.java) ~[na:na]
at org.opendaylight.yang.gen.v1.urn.opendaylight.flow.service.rev130819.SalFlowService$$Broker$DirectProxy.addFlow(SalFlowService$$Broker$DirectProxy.java) ~[na:na]
at org.opendaylight.controller.frm.flow.FlowTransaction.add(FlowTransaction.java:121) ~[na:na]
at org.opendaylight.controller.frm.AbstractTransaction.callRpcs(AbstractTransaction.java:63) ~[na:na]
at org.opendaylight.controller.frm.AbstractTransaction.finish(AbstractTransaction.java:35) ~[na:na]
at org.opendaylight.controller.md.sal.common.impl.service.TwoPhaseCommit.call(TwoPhaseCommit.java:110) [bundlefile:na]
at org.opendaylight.controller.md.sal.common.impl.service.TwoPhaseCommit.call(TwoPhaseCommit.java:38) [bundlefile:na]
at java.util.concurrent.FutureTask.run(FutureTask.java:262) [na:1.7.0_51]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_51]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_51]
at java.lang.Thread.run(Thread.java:744) [na:1.7.0_51
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

2. We have also observed address not found issue, when the rpc cache contains all rpc registrations from test-provider bundle. RPC cache also contains all registrations from ModelDrivenSwitch except for flow and group. In this scenario: this.supportedRpcs = mappingService.getRpcQNamesFor(service); in BindingIndependentConnector.java::DomToBindingRpcForwarder(Class<? extends RpcService> service, Class<? extends BaseIdentity> context) does not return empty set and contains correct rpcs. We have also observed that, in this scenario, if you have multiple test-provider bundles, only the first registration is available in rpc-cache.

3. In third scenario, if the switch is up and running before the controller is started. By the time the controller is fully up, none of the rpc registrations from ModelDrivenSwitch go to rpc-cache.