[CONTROLLER-1429] Clustering: Post reconvergance (reboot of leader) in 5 node cluster, Flows not installed (in operational data store) even pushed via both leader or follower

Thu, 18 Feb 2016 12:12:36 +0000

After initial startup if we reboot leader node, post convergance Flows are not installed in the switch even if flow pushed from new leader or follower node.
Inventory crawler shows flow in config data store but not in operational datastore.

1.
Started a 5 node cluster with below IPs. deploy_odl.py script is used to start it.
[
controller-1 = 10.183.181.41
controller-2 = 10.183.181.42
controller-3 = 10.183.181.43
controller-4 = 10.183.181.44
controller-5 = 10.183.181.45
]

cd /home/mininet/integration/test/tools/clustering/cluster-deployer/

root@mininet-vm:/home/mininet/integration/test/tools/clustering/cluster-deployer#./deploy_odl.py --clean --distribution=/home/mininet/controller-Li/distribution-karaf-0.3.1-Lithium-SR1.zip --rootdir=/home/mininet/controller-Li --hosts=10.183.181.41,10.183.181.42,10.183.181.43,10.183.181.44,10.183.181.45 --user=root --password=rootroot --template=/multi-node-test

2. I observe in all the cluster nodes 2551 and 2550 ports came up properly.

root@mininet-vm:~# netstat -na tcp6 0 0 10.183.181.41:2550 tcp6 0 0 10.183.181.41:48109 tcp6 0 0 10.183.181.41:42152 tcp6 0 0 10.183.181.41:2550 tcp6 0 0 10.183.181.41:34346 tcp6 0 0 10.183.181.41:36484 tcp6 0 0 10.183.181.41:2550 tcp6 0 0 10.183.181.41:2550 tcp6 0 0 10.183.181.41:2550 root@mininet-vm:~# netstat -na tcp6 0 0 10.183.181.41:2551 tcp6 0 0 10.183.181.41:2551 tcp6 0 0 10.183.181.41:48196 tcp6 0 0 10.183.181.41:2551 tcp6 0 0 10.183.181.41:53302 tcp6 0 0 10.183.181.41:56779 tcp6 0 0 10.183.181.41:2551 tcp6 0 0 10.183.181.41:2551 tcp6 0 0 10.183.181.41:53763 root@mininet-vm:~# | grep 2550
:::* LISTEN
10.183.181.44:2550 ESTABLISHED
10.183.181.45:2550 ESTABLISHED
10.183.181.44:44754 ESTABLISHED
10.183.181.43:2550 ESTABLISHED
10.183.181.42:2550 ESTABLISHED
10.183.181.43:51115 ESTABLISHED
10.183.181.45:45284 ESTABLISHED
10.183.181.42:50675 ESTABLISHED
| grep 2551
:::* LISTEN
10.183.181.45:33739 ESTABLISHED
10.183.181.43:2551 ESTABLISHED
10.183.181.42:55729 ESTABLISHED
10.183.181.45:2551 ESTABLISHED
10.183.181.42:2551 ESTABLISHED
10.183.181.43:36887 ESTABLISHED
10.183.181.44:45111 ESTABLISHED
10.183.181.44:2551 ESTABLISHED

3. from jconsole i observe controller-2 is leader. I connect the mininet switch to follower controller-1 and pushed flows from controller-3, i observer all flows are configured in switch.

sdnc-host2:~ # c3
Welcome to Ubuntu 14.04.1 LTS (GNU/Linux 3.13.0-43-generic x86_64)

Documentation: https://help.ubuntu.com/

Last login: Thu Oct 15 23:33:41 2015 from 10.183.181.12
root@mininet-vm:~# cd /home/mininet/integration/test/tools/odl-mdsal-clustering- tests/clustering-performance-test
root@mininet-vm:/home/mininet/integration/test/tools/odl-mdsal-clustering-tests/ clustering-performance-test# ./flow_config_blaster.py --flows 10 --auth --no-del ete

Cycle 0:
Thread 0:
Adding 10 flows on 1 nodes

Thread 0 results (ADD):
Elapsed time: 1.22s,
Requests/s: 8.23 OK, 8.23 Total
Flows/s: 8.23 OK, 8.23 Total
Stats (

{Requests}, {Flows}): {200: 0, 204: 10} {200: 0, 204: 10}

*** Test summary:
Elapsed time: 1.83s
Peak requests/s: 8.23 OK, 8.23 Total
Peak flows/s: 8.23 OK, 8.23 Total
Avg. requests/s: 5.46 OK, 5.46 Total (66.33% of peak total)
Avg. flows/s: 5.46 OK, 5.46 Total (66.33% of peak total)

*** Total flows added: 10
HTTP[OK] results: 10

root@mininet-vm:~# sudo mn --controller=remote,ip=10.183.181.41 --topo=linear,1 --switch ovsk,protocols=OpenFlow13
*** Creating network
*** Adding controller
*** Adding hosts:
h1
*** Adding switches:
have protcol params!
s1
*** Adding links:
(h1, s1)
*** Configuring hosts
h1
*** Starting controller
*** Starting 1 switches
s1 OVSswitch opts: protocols=OpenFlow13

*** Starting CLI:
mininet> sh ovs-ofctl -O Openflow13 dump-flows s1
OFPST_FLOW reply (OF1.3) (xid=0x2):
cookie=0x6, duration=19.436s, table=0, n_packets=0, n_bytes=0, idle_timeout=650 00, hard_timeout=65000, priority=2,ip,nw_dst=10.0.0.7 actions=drop
cookie=0x5, duration=19.428s, table=0, n_packets=0, n_bytes=0, idle_timeout=650 00, hard_timeout=65000, priority=2,ip,nw_dst=10.0.0.6 actions=drop
cookie=0x1, duration=19.432s, table=0, n_packets=0, n_bytes=0, idle_timeout=650 00, hard_timeout=65000, priority=2,ip,nw_dst=10.0.0.2 actions=drop
cookie=0x8, duration=19.426s, table=0, n_packets=0, n_bytes=0, idle_timeout=650 00, hard_timeout=65000, priority=2,ip,nw_dst=10.0.0.9 actions=drop
cookie=0x3, duration=19.426s, table=0, n_packets=0, n_bytes=0, idle_timeout=650 00, hard_timeout=65000, priority=2,ip,nw_dst=10.0.0.4 actions=drop
cookie=0x9, duration=19.426s, table=0, n_packets=0, n_bytes=0, idle_timeout=650 00, hard_timeout=65000, priority=2,ip,nw_dst=10.0.0.10 actions=drop
cookie=0x7, duration=19.435s, table=0, n_packets=0, n_bytes=0, idle_timeout=650 00, hard_timeout=65000, priority=2,ip,nw_dst=10.0.0.8 actions=drop
cookie=0x0, duration=19.438s, table=0, n_packets=0, n_bytes=0, idle_timeout=650 00, hard_timeout=65000, priority=2,ip,nw_dst=10.0.0.1 actions=drop
cookie=0x4, duration=19.434s, table=0, n_packets=0, n_bytes=0, idle_timeout=650 00, hard_timeout=65000, priority=2,ip,nw_dst=10.0.0.5 actions=drop
cookie=0x2, duration=19.428s, table=0, n_packets=0, n_bytes=0, idle_timeout=650 00, hard_timeout=65000, priority=2,ip,nw_dst=10.0.0.3 actions=drop
mininet> sh ovs-ofctl -O Openflow13 dump-flows s1
OFPST_FLOW reply (OF1.3) (xid=0x2):
mininet>
mininet>

Similarly i tried connecting the switch in various other follower node and pushed flows from all nodes , i find flows are installed.

4. I rebooted the controller-2 which is leader, from jconsoe i observe now controller-1 is the new leader.
Now I connect switch to follower controller-5 and push flows from another follower node controller-3, it does not work, i donot see flows are pushed to switch.

root@mininet-vm:/home/mininet/integration/test/tools/odl-mdsal-clustering-tests/clustering-performance-test# ./flow_config_blaster.py --flows 15 --auth --no-delete

Cycle 0:
Thread 0:
Adding 15 flows on 1 nodes

Thread 0 results (ADD):
Elapsed time: 0.43s,
Requests/s: 34.68 OK, 34.68 Total
Flows/s: 34.68 OK, 34.68 Total
Stats ({Requests}

{Flows}

{200: 0, 204: 15}

- - Test summary:
    Elapsed time: 0.51s
    Peak requests/s: 34.68 OK, 34.68 Total
    Peak flows/s: 34.68 OK, 34.68 Total
    Avg. requests/s: 29.27 OK, 29.27 Total (84.40% of peak total)
    Avg. flows/s: 29.27 OK, 29.27 Total (84.40% of peak total)

- - Total flows added: 15
    HTTP[OK] results: 15

root@mininet-vm:~# sudo mn --controller=remote,ip=10.183.181.45 --topo=linear,1 --switch ovsk,protocols=OpenFlow13

- - Creating network
  - Adding controller
  - Adding hosts:
    h1
  - Adding switches:
    have protcol params!
    s1
  - Adding links:
    (h1, s1)
  - Configuring hosts
    h1
  - Starting controller
  - Starting 1 switches
    s1 OVSswitch opts: protocols=OpenFlow13

- - Starting CLI:
    mininet> sh ovs-ofctl -O Openflow13 dump-flows s1
    OFPST_FLOW reply (OF1.3) (xid=0x2):

Inventory crawler shows flow in config data store but not in operational datastore.

root@mininet-vm:/home/mininet/integration/test/tools/odl-mdsal-clustering-tests/clustering-performance-test# ./inventory_crawler.py --auth --host 10.183.181.43 --datastore operational
Crawling 'http://10.183.181.43:8181/restconf/operational/opendaylight-inventory:nodes'

Totals:
Nodes: 1
Reported flows: 9
Found flows: 0
root@mininet-vm:/home/mininet/integration/test/tools/odl-mdsal-clustering-tests/clustering-performance-test# ./inventory_crawler.py --auth --host 10.183.181.43 --datastore config
Crawling 'http://10.183.181.43:8181/restconf/config/opendaylight-inventory:nodes'

Totals:
Nodes: 1
Reported flows: 0
Found flows: 15

Similarly i tried to connect the switch to the leader and pushed from new leader controller-1, i find flows are not pushed to switch.

Please find attached logs of all controller nodes.

OpenDaylight JIRA

[CONTROLLER-1429] Clustering: Post reconvergance (reboot of leader) in 5 node cluster, Flows not installed (in operational data store) even pushed via both leader or follower