ODL TR.rar
[CONTROLLER-1429] Clustering: Post reconvergance (reboot of leader) in 5 node cluster, Flows not installed (in operational data store) even pushed via both leader or follower Created: 16/Oct/15 Updated: 18/Feb/16 Resolved: 18/Feb/16 |
|
| Status: | Resolved |
| Project: | controller |
| Component/s: | clustering |
| Affects Version/s: | Lithium |
| Fix Version/s: | None |
| Type: | Bug | ||
| Reporter: | Sanjib Mohapatra | Assignee: | Moiz Raja |
| Resolution: | Cannot Reproduce | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Environment: |
Operating System: All |
||
| Attachments: |
ODL TR.rar
|
| External issue ID: | 4484 |
| Priority: | Normal |
| Description |
|
After initial startup if we reboot leader node, post convergance Flows are not installed in the switch even if flow pushed from new leader or follower node. 1. cd /home/mininet/integration/test/tools/clustering/cluster-deployer/ root@mininet-vm:/home/mininet/integration/test/tools/clustering/cluster-deployer#./deploy_odl.py --clean --distribution=/home/mininet/controller-Li/distribution-karaf-0.3.1-Lithium-SR1.zip --rootdir=/home/mininet/controller-Li --hosts=10.183.181.41,10.183.181.42,10.183.181.43,10.183.181.44,10.183.181.45 --user=root --password=rootroot --template=/multi-node-test 2. I observe in all the cluster nodes 2551 and 2550 ports came up properly. root@mininet-vm:~# netstat -na | grep 2550 3. from jconsole i observe controller-2 is leader. I connect the mininet switch to follower controller-1 and pushed flows from controller-3, i observer all flows are configured in switch. sdnc-host2:~ # c3
Last login: Thu Oct 15 23:33:41 2015 from 10.183.181.12 Cycle 0: Thread 0 results (ADD): *** Test summary: Elapsed time: 1.83s Peak requests/s: 8.23 OK, 8.23 Total Peak flows/s: 8.23 OK, 8.23 Total Avg. requests/s: 5.46 OK, 5.46 Total (66.33% of peak total) Avg. flows/s: 5.46 OK, 5.46 Total (66.33% of peak total) *** Total flows added: 10 HTTP[OK] results: 10 root@mininet-vm:~# sudo mn --controller=remote,ip=10.183.181.41 --topo=linear,1 --switch ovsk,protocols=OpenFlow13 *** Creating network *** Adding controller *** Adding hosts: h1 *** Adding switches: have protcol params! s1 *** Adding links: (h1, s1) *** Configuring hosts h1 *** Starting controller *** Starting 1 switches s1 OVSswitch opts: protocols=OpenFlow13 *** Starting CLI: mininet> sh ovs-ofctl -O Openflow13 dump-flows s1 OFPST_FLOW reply (OF1.3) (xid=0x2): cookie=0x6, duration=19.436s, table=0, n_packets=0, n_bytes=0, idle_timeout=650 00, hard_timeout=65000, priority=2,ip,nw_dst=10.0.0.7 actions=drop cookie=0x5, duration=19.428s, table=0, n_packets=0, n_bytes=0, idle_timeout=650 00, hard_timeout=65000, priority=2,ip,nw_dst=10.0.0.6 actions=drop cookie=0x1, duration=19.432s, table=0, n_packets=0, n_bytes=0, idle_timeout=650 00, hard_timeout=65000, priority=2,ip,nw_dst=10.0.0.2 actions=drop cookie=0x8, duration=19.426s, table=0, n_packets=0, n_bytes=0, idle_timeout=650 00, hard_timeout=65000, priority=2,ip,nw_dst=10.0.0.9 actions=drop cookie=0x3, duration=19.426s, table=0, n_packets=0, n_bytes=0, idle_timeout=650 00, hard_timeout=65000, priority=2,ip,nw_dst=10.0.0.4 actions=drop cookie=0x9, duration=19.426s, table=0, n_packets=0, n_bytes=0, idle_timeout=650 00, hard_timeout=65000, priority=2,ip,nw_dst=10.0.0.10 actions=drop cookie=0x7, duration=19.435s, table=0, n_packets=0, n_bytes=0, idle_timeout=650 00, hard_timeout=65000, priority=2,ip,nw_dst=10.0.0.8 actions=drop cookie=0x0, duration=19.438s, table=0, n_packets=0, n_bytes=0, idle_timeout=650 00, hard_timeout=65000, priority=2,ip,nw_dst=10.0.0.1 actions=drop cookie=0x4, duration=19.434s, table=0, n_packets=0, n_bytes=0, idle_timeout=650 00, hard_timeout=65000, priority=2,ip,nw_dst=10.0.0.5 actions=drop cookie=0x2, duration=19.428s, table=0, n_packets=0, n_bytes=0, idle_timeout=650 00, hard_timeout=65000, priority=2,ip,nw_dst=10.0.0.3 actions=drop mininet> sh ovs-ofctl -O Openflow13 dump-flows s1 OFPST_FLOW reply (OF1.3) (xid=0x2): mininet> mininet> Similarly i tried connecting the switch in various other follower node and pushed flows from all nodes , i find flows are installed. 4. I rebooted the controller-2 which is leader, from jconsoe i observe now controller-1 is the new leader. Now I connect switch to follower controller-5 and push flows from another follower node controller-3, it does not work, i donot see flows are pushed to switch. root@mininet-vm:/home/mininet/integration/test/tools/odl-mdsal-clustering-tests/clustering-performance-test# ./flow_config_blaster.py --flows 15 --auth --no-delete Cycle 0: Thread 0: Adding 15 flows on 1 nodes Thread 0 results (ADD): Elapsed time: 0.43s, Requests/s: 34.68 OK, 34.68 Total Flows/s: 34.68 OK, 34.68 Total Stats ({Requests} , {Flows}): {200: 0, 204: 15}{200: 0, 204: 15}
root@mininet-vm:~# sudo mn --controller=remote,ip=10.183.181.45 --topo=linear,1 --switch ovsk,protocols=OpenFlow13
Inventory crawler shows flow in config data store but not in operational datastore. root@mininet-vm:/home/mininet/integration/test/tools/odl-mdsal-clustering-tests/clustering-performance-test# ./inventory_crawler.py --auth --host 10.183.181.43 --datastore operational Totals: Totals: Similarly i tried to connect the switch to the leader and pushed from new leader controller-1, i find flows are not pushed to switch. Please find attached logs of all controller nodes. |
| Comments |
| Comment by Sanjib Mohapatra [ 16/Oct/15 ] |
|
Attachment ODL TR.rar has been added with description: karaf logs of all the controller nodes |
| Comment by Moiz Raja [ 27/Oct/15 ] |
|
This is not supported in Lithium SR1. Please try on Lithium SR2. |
| Comment by Sanjib Mohapatra [ 03/Nov/15 ] |
|
I have tested in SR2, 9 out of 10 times i observe the issue. The issue still exists in it. |
| Comment by Tom Pantelis [ 18/Feb/16 ] |
|
This was likely due to the routed RPC registration timing issue that has since been addressed. Since OF clustering seems to be working fine with recent fixes/testing in Be, I'm closing this bug. |