Clustering: Post reconvergance (reboot of leader) in 5 node cluster, Flows not installed (in operational data store) even pushed via both leader or follower

After initial startup if we reboot leader node, post convergance Flows are not installed in the switch even if flow pushed from new leader or follower node.
Inventory crawler shows flow in config data store but not in operational datastore.

1.
Started a 5 node cluster with below IPs. deploy_odl.py script is used to start it.
[
controller-1 = 10.183.181.41
controller-2 = 10.183.181.42
controller-3 = 10.183.181.43
controller-4 = 10.183.181.44
controller-5 = 10.183.181.45
]

cd /home/mininet/integration/test/tools/clustering/cluster-deployer/

root@mininet-vm:/home/mininet/integration/test/tools/clustering/cluster-deployer#./deploy_odl.py --clean --distribution=/home/mininet/controller-Li/distribution-karaf-0.3.1-Lithium-SR1.zip --rootdir=/home/mininet/controller-Li --hosts=10.183.181.41,10.183.181.42,10.183.181.43,10.183.181.44,10.183.181.45 --user=root --password=rootroot --template=/multi-node-test

2. I observe in all the cluster nodes 2551 and 2550 ports came up properly.

root@mininet-vm:~# netstat -na | grep 2550
tcp6 0 0 10.183.181.41:2550 :::* LISTEN
tcp6 0 0 10.183.181.41:48109 10.183.181.44:2550 ESTABLISHED
tcp6 0 0 10.183.181.41:42152 10.183.181.45:2550 ESTABLISHED
tcp6 0 0 10.183.181.41:2550 10.183.181.44:44754 ESTABLISHED
tcp6 0 0 10.183.181.41:34346 10.183.181.43:2550 ESTABLISHED
tcp6 0 0 10.183.181.41:36484 10.183.181.42:2550 ESTABLISHED
tcp6 0 0 10.183.181.41:2550 10.183.181.43:51115 ESTABLISHED
tcp6 0 0 10.183.181.41:2550 10.183.181.45:45284 ESTABLISHED
tcp6 0 0 10.183.181.41:2550 10.183.181.42:50675 ESTABLISHED
root@mininet-vm:~# netstat -na | grep 2551
tcp6 0 0 10.183.181.41:2551 :::* LISTEN
tcp6 0 0 10.183.181.41:2551 10.183.181.45:33739 ESTABLISHED
tcp6 0 0 10.183.181.41:48196 10.183.181.43:2551 ESTABLISHED
tcp6 0 0 10.183.181.41:2551 10.183.181.42:55729 ESTABLISHED
tcp6 0 0 10.183.181.41:53302 10.183.181.45:2551 ESTABLISHED
tcp6 0 0 10.183.181.41:56779 10.183.181.42:2551 ESTABLISHED
tcp6 0 0 10.183.181.41:2551 10.183.181.43:36887 ESTABLISHED
tcp6 0 0 10.183.181.41:2551 10.183.181.44:45111 ESTABLISHED
tcp6 0 0 10.183.181.41:53763 10.183.181.44:2551 ESTABLISHED
root@mininet-vm:~#

3. from jconsole i observe controller-2 is leader. I connect the mininet switch to follower controller-1 and pushed flows from controller-3, i observer all flows are configured in switch.

sdnc-host2:~ # c3
Welcome to Ubuntu 14.04.1 LTS (GNU/Linux 3.13.0-43-generic x86_64)

• Documentation: https://help.ubuntu.com/

Last login: Thu Oct 15 23:33:41 2015 from 10.183.181.12
root@mininet-vm:~# cd /home/mininet/integration/test/tools/odl-mdsal-clustering- tests/clustering-performance-test
root@mininet-vm:/home/mininet/integration/test/tools/odl-mdsal-clustering-tests/ clustering-performance-test# ./flow_config_blaster.py --flows 10 --auth --no-del ete

Cycle 0:
Thread 0:
Adding 10 flows on 1 nodes

Thread 0 results (ADD):
Elapsed time: 1.22s,
Requests/s: 8.23 OK, 8.23 Total
Flows/s: 8.23 OK, 8.23 Total
Stats (

{Requests}, {Flows}): {200: 0, 204: 10} {200: 0, 204: 10}

*** Test summary:
Elapsed time: 1.83s
Peak requests/s: 8.23 OK, 8.23 Total
Peak flows/s: 8.23 OK, 8.23 Total
Avg. requests/s: 5.46 OK, 5.46 Total (66.33% of peak total)
Avg. flows/s: 5.46 OK, 5.46 Total (66.33% of peak total)

*** Total flows added: 10
HTTP[OK] results: 10


root@mininet-vm:~# sudo mn --controller=remote,ip=10.183.181.41 --topo=linear,1 --switch ovsk,protocols=OpenFlow13
*** Creating network
*** Adding controller
*** Adding hosts:
h1
*** Adding switches:
have protcol params!
s1
*** Adding links:
(h1, s1)
*** Configuring hosts
h1
*** Starting controller
*** Starting 1 switches
s1 OVSswitch opts: protocols=OpenFlow13

*** Starting CLI:
mininet> sh ovs-ofctl -O Openflow13 dump-flows s1
OFPST_FLOW reply (OF1.3) (xid=0x2):
cookie=0x6, duration=19.436s, table=0, n_packets=0, n_bytes=0, idle_timeout=650 00, hard_timeout=65000, priority=2,ip,nw_dst=10.0.0.7 actions=drop
cookie=0x5, duration=19.428s, table=0, n_packets=0, n_bytes=0, idle_timeout=650 00, hard_timeout=65000, priority=2,ip,nw_dst=10.0.0.6 actions=drop
cookie=0x1, duration=19.432s, table=0, n_packets=0, n_bytes=0, idle_timeout=650 00, hard_timeout=65000, priority=2,ip,nw_dst=10.0.0.2 actions=drop
cookie=0x8, duration=19.426s, table=0, n_packets=0, n_bytes=0, idle_timeout=650 00, hard_timeout=65000, priority=2,ip,nw_dst=10.0.0.9 actions=drop
cookie=0x3, duration=19.426s, table=0, n_packets=0, n_bytes=0, idle_timeout=650 00, hard_timeout=65000, priority=2,ip,nw_dst=10.0.0.4 actions=drop
cookie=0x9, duration=19.426s, table=0, n_packets=0, n_bytes=0, idle_timeout=650 00, hard_timeout=65000, priority=2,ip,nw_dst=10.0.0.10 actions=drop
cookie=0x7, duration=19.435s, table=0, n_packets=0, n_bytes=0, idle_timeout=650 00, hard_timeout=65000, priority=2,ip,nw_dst=10.0.0.8 actions=drop
cookie=0x0, duration=19.438s, table=0, n_packets=0, n_bytes=0, idle_timeout=650 00, hard_timeout=65000, priority=2,ip,nw_dst=10.0.0.1 actions=drop
cookie=0x4, duration=19.434s, table=0, n_packets=0, n_bytes=0, idle_timeout=650 00, hard_timeout=65000, priority=2,ip,nw_dst=10.0.0.5 actions=drop
cookie=0x2, duration=19.428s, table=0, n_packets=0, n_bytes=0, idle_timeout=650 00, hard_timeout=65000, priority=2,ip,nw_dst=10.0.0.3 actions=drop
mininet> sh ovs-ofctl -O Openflow13 dump-flows s1
OFPST_FLOW reply (OF1.3) (xid=0x2):
mininet>
mininet>

Similarly i tried connecting the switch in various other follower node and pushed flows from all nodes , i find flows are installed.

4. I rebooted the controller-2 which is leader, from jconsoe i observe now controller-1 is the new leader.
Now I connect switch to follower controller-5 and push flows from another follower node controller-3, it does not work, i donot see flows are pushed to switch.

root@mininet-vm:/home/mininet/integration/test/tools/odl-mdsal-clustering-tests/clustering-performance-test# ./flow_config_blaster.py --flows 15 --auth --no-delete

Cycle 0:
Thread 0:
Adding 15 flows on 1 nodes

Thread 0 results (ADD):
Elapsed time: 0.43s,
Requests/s: 34.68 OK, 34.68 Total
Flows/s: 34.68 OK, 34.68 Total
Stats ({Requests}

,

{Flows}

):

{200: 0, 204: 15}

{200: 0, 204: 15}

• • • Test summary:
      Elapsed time: 0.51s
      Peak requests/s: 34.68 OK, 34.68 Total
      Peak flows/s: 34.68 OK, 34.68 Total
      Avg. requests/s: 29.27 OK, 29.27 Total (84.40% of peak total)
      Avg. flows/s: 29.27 OK, 29.27 Total (84.40% of peak total)

• • • Total flows added: 15
      HTTP[OK] results: 15

root@mininet-vm:~# sudo mn --controller=remote,ip=10.183.181.45 --topo=linear,1 --switch ovsk,protocols=OpenFlow13

• • • Creating network
    • Adding controller
    • Adding hosts:
      h1
    • Adding switches:
      have protcol params!
      s1
    • Adding links:
      (h1, s1)
    • Configuring hosts
      h1
    • Starting controller
    • Starting 1 switches
      s1 OVSswitch opts: protocols=OpenFlow13

• • • Starting CLI:
      mininet> sh ovs-ofctl -O Openflow13 dump-flows s1
      OFPST_FLOW reply (OF1.3) (xid=0x2):

Inventory crawler shows flow in config data store but not in operational datastore.

root@mininet-vm:/home/mininet/integration/test/tools/odl-mdsal-clustering-tests/clustering-performance-test# ./inventory_crawler.py --auth --host 10.183.181.43 --datastore operational
Crawling 'http://10.183.181.43:8181/restconf/operational/opendaylight-inventory:nodes'

Totals:
Nodes: 1
Reported flows: 9
Found flows: 0
root@mininet-vm:/home/mininet/integration/test/tools/odl-mdsal-clustering-tests/clustering-performance-test# ./inventory_crawler.py --auth --host 10.183.181.43 --datastore config
Crawling 'http://10.183.181.43:8181/restconf/config/opendaylight-inventory:nodes'

Totals:
Nodes: 1
Reported flows: 0
Found flows: 15

Similarly i tried to connect the switch to the leader and pushed from new leader controller-1, i find flows are not pushed to switch.

Please find attached logs of all controller nodes.