Problems observed when connecting openflow switches to multiple instances of controller

If openflow switches are connected to multiple instances of the clustered controller the following problems were observed,

1. Sometimes all switches do not show up in inventory. This is because on all nodes in the controller inventory manager is active and trying to add the same set of nodes to inventory. If the switch was connected to 3 different controller instances then on atleast 2 of those instances the transaction to add nodes to inventory will fail. This further complicated by the fact that inventory manager batches changes. So for example on instance 1 inventory manager may batch changes which adds switch 1 and 2 and on instance 2 it may batch changes to add switch 2 and 3. Now let's say the transaction on instance 1 succeeds, this will cause the transaction on instance 2 to fail and switch 3 will thus not appear in inventory. This will be a problem if I had chosen to connect switch 2 to controller instance 1 and 2 and switch to controller instance 2 and 3.

2. On all instance of the controller statistics manager is active this causes duplicate statistics collection for each switch that is connected to multiple instances of the controller which puts additional load on the switch and causes a lot of data transaction failures - making statistics collection fragile and useless.

3. If a switch is connected to two controller instances and if the controller instances were equal then packetins will be received by both the controller instance. Applications which have been created to program flows based on packetins will thus attempt to do the same thing on both controller instances. This has the potential for corrupting the flow table.

4. When a switch is connected to multiple instances of the controller let's say instance 1 and 2 only one of those instances would have a successful rpc registration for that switch. This is because the remote rpc connector chooses the last registered rpc. Now if the switch first connnected to instance 1 and then to instance 2 it us likely that the add flow rpc for that switch was last registered on instance 2. In this scenario if you were to bring down instance 2 of the controller - all flow programming for that switch will cease to work because now the rpc registration held by the remote rpc connector is invalid.

To make all these problems (and possibly many more) go away the openflowplugin needs to ensure that only one instance of the plugin controls a switch.