[NETVIRT-1053]  stale entries not getting removed on ACL table 212 and 242 Created: 04/Jan/18  Updated: 18/Jun/18  Resolved: 28/Mar/18

Status: Verified
Project: netvirt
Component/s: None
Affects Version/s: Oxygen
Fix Version/s: None

Type: Bug Priority: High
Reporter: RajaRajan Manickam Assignee: RajaRajan Manickam
Resolution: Cannot Reproduce Votes: 0
Labels: VPNV6
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Priority: High

 Description   

Description:

stale entries not getting removed on ACL table 212 and 242 on RC3 Nitrogen build

Steps:

1. Create Network,subnet,port,router and do and basic testing.
2. Remove all network,subnet,port and router and check all stale entries removed.

Result:

Stale entries not removed properly.

Logs:

Devstack Output:

stack@rajarajan-14:~$ neutron net-list

stack@rajarajan-14:~$ neutron subnet-list

stack@rajarajan-14:~$ neutron port-list

stack@rajarajan-14:~$ neutron router-list

stack@rajarajan-14:~$
stack@rajarajan-14:~/devstack$ sudo ovs-ofctl -OOpenflow13 dump-flows br-int | more
OFPST_FLOW reply (OF1.3) (xid=0x2):
 cookie=0x8000001, duration=4215.174s, table=0, n_packets=5794, n_bytes=419197, priority=5,in_port=1 actions=write_metadata:0x20000000001/0xfffff0000000001,goto_table:36
 cookie=0x8000000, duration=4215.501s, table=17, n_packets=0, n_bytes=0, priority=0,metadata=0x8000000000000000/0xf000000000000000 actions=write_metadata:0x9000000000000000/0xf00000
0000000000,goto_table:80
 cookie=0x6800000, duration=4215.377s, table=18, n_packets=0, n_bytes=0, priority=0 actions=goto_table:38
 cookie=0x8220016, duration=4215.174s, table=19, n_packets=0, n_bytes=0, priority=100,arp,arp_op=2 actions=resubmit(,17)
 cookie=0x8220015, duration=4215.174s, table=19, n_packets=149, n_bytes=6258, priority=100,arp,arp_op=1 actions=resubmit(,17)
 cookie=0x1080000, duration=4215.174s, table=19, n_packets=24, n_bytes=2008, priority=0 actions=resubmit(,17)
 cookie=0x1030000, duration=4215.174s, table=20, n_packets=0, n_bytes=0, priority=0 actions=goto_table:80
 cookie=0x8000004, duration=4215.174s, table=22, n_packets=0, n_bytes=0, priority=0 actions=CONTROLLER:65535
 cookie=0x1080000, duration=4215.174s, table=23, n_packets=0, n_bytes=0, priority=0 actions=resubmit(,17)
 cookie=0x1080000, duration=4215.174s, table=24, n_packets=0, n_bytes=0, priority=0 actions=resubmit(,17)
 cookie=0x822002d, duration=4214.738s, table=43, n_packets=282, n_bytes=11844, priority=100,arp,arp_op=1 actions=group:5001
 cookie=0x822002e, duration=4214.738s, table=43, n_packets=3, n_bytes=126, priority=100,arp,arp_op=2 actions=CONTROLLER:65535,resubmit(,48)
 cookie=0x8220000, duration=4215.258s, table=43, n_packets=56, n_bytes=6728, priority=0 actions=goto_table:48
 cookie=0x4000000, duration=4215.377s, table=45, n_packets=264, n_bytes=15232, priority=0 actions=resubmit(,17)
 cookie=0x8500000, duration=4215.377s, table=48, n_packets=341, n_bytes=18698, priority=0 actions=resubmit(,49),resubmit(,50)
 cookie=0x8050001, duration=4215.377s, table=50, n_packets=0, n_bytes=0, priority=10,reg4=0x1 actions=goto_table:51
 cookie=0x8050000, duration=4215.377s, table=50, n_packets=0, n_bytes=0, priority=0 actions=CONTROLLER:65535,learn(table=49,hard_timeout=10,priority=0,cookie=0x8600000,NXM_OF_ETH_SR
C[],NXM_NX_REG1[0..19],load:0x1->NXM_NX_REG4[0..7]),goto_table:51
 cookie=0x8030000, duration=4215.174s, table=51, n_packets=0, n_bytes=0, priority=15,dl_dst=01:80:c2:00:00:00/ff:ff:ff:ff:ff:f0 actions=drop
 cookie=0x8030000, duration=4215.377s, table=51, n_packets=465, n_bytes=23830, priority=0 actions=goto_table:52
 cookie=0x6800000, duration=4215.377s, table=60, n_packets=329, n_bytes=15674, priority=0 actions=resubmit(,17)
 cookie=0x1030000, duration=4215.174s, table=80, n_packets=0, n_bytes=0, priority=0 actions=resubmit(,17)
 cookie=0x8220000, duration=4215.174s, table=81, n_packets=280, n_bytes=11760, priority=0 actions=drop
 cookie=0x4000001, duration=4215.502s, table=90, n_packets=0, n_bytes=0, priority=0 actions=resubmit(,17)
 cookie=0x6900000, duration=4215.029s, table=211, n_packets=0, n_bytes=0, priority=63009,arp actions=drop
 cookie=0x6900000, duration=4215.029s, table=211, n_packets=1, n_bytes=342, priority=61009,ip actions=drop
 cookie=0x6900000, duration=4214.896s, table=211, n_packets=15, n_bytes=1318, priority=61009,ipv6 actions=drop
 cookie=0x6900000, duration=4215.377s, table=211, n_packets=0, n_bytes=0, priority=0 actions=drop
 cookie=0x6900000, duration=3781.895s, table=212, n_packets=0, n_bytes=0, priority=50,ipv6,metadata=0x138a000000/0xffff000000,ipv6_dst=fe80::f816:3eff:fe5f:68b9 actions=write_metada
ta:0x2/0xfffffe,goto_table:213
 cookie=0x6900000, duration=3781.702s, table=212, n_packets=0, n_bytes=0, priority=50,ipv6,metadata=0x138a000000/0xffff000000,ipv6_dst=2001:db8:1234:0:f816:3eff:fe5f:68b9 actions=wr
ite_metadata:0x2/0xfffffe,goto_table:213
 cookie=0x6900000, duration=3740.693s, table=212, n_packets=0, n_bytes=0, priority=50,ipv6,metadata=0x138a000000/0xffff000000,ipv6_dst=2001:db8:1234:0:f816:3eff:fefe:71f1 actions=wr
ite_metadata:0x2/0xfffffe,goto_table:213
 cookie=0x6900000, duration=3740.244s, table=212, n_packets=0, n_bytes=0, priority=50,ipv6,metadata=0x138a000000/0xffff000000,ipv6_dst=fe80::f816:3eff:fefe:71f1 actions=write_metada
ta:0x2/0xfffffe,goto_table:213
 cookie=0x6900000, duration=3781.819s, table=212, n_packets=2, n_bytes=196, priority=50,ip,metadata=0x138a000000/0xffff000000,nw_dst=10.1.1.11 actions=write_metadata:0x2/0xfffffe,go
to_table:213
 cookie=0x6900000, duration=3740.481s, table=212, n_packets=2, n_bytes=196, priority=50,ip,metadata=0x138a000000/0xffff000000,nw_dst=10.1.1.14 actions=write_metadata:0x2/0xfffffe,go
to_table:213
 cookie=0x6900000, duration=3661.408s, table=212, n_packets=0, n_bytes=0, priority=50,ip,metadata=0x30d40/0xfffffe,nw_dst=10.1.1.5 actions=write_metadata:0x2/0xfffffe,goto_table:213
 cookie=0x6900000, duration=3659.933s, table=212, n_packets=0, n_bytes=0, priority=50,ip,metadata=0x30d40/0xfffffe,nw_dst=10.1.1.6 actions=write_metadata:0x2/0xfffffe,goto_table:213
 cookie=0x6900000, duration=3660.861s, table=212, n_packets=0, n_bytes=0, priority=50,ipv6,metadata=0x30d40/0xfffffe,ipv6_dst=fe80::f816:3eff:fece:2cc5 actions=write_metadata:0x2/0x
fffffe,goto_table:213
 cookie=0x6900000, duration=3660.535s, table=212, n_packets=0, n_bytes=0, priority=50,ipv6,metadata=0x30d40/0xfffffe,ipv6_dst=2001:db8:1234:0:f816:3eff:fece:2cc5 actions=write_metad
ata:0x2/0xfffffe,goto_table:213
 cookie=0x6900000, duration=3660s, table=212, n_packets=0, n_bytes=0, priority=50,ipv6,metadata=0x30d40/0xfffffe,ipv6_dst=fe80::f816:3eff:fef5:5b21 actions=write_metadata:0x2/0xffff
fe,goto_table:213
 cookie=0x6900000, duration=3659.817s, table=212, n_packets=0, n_bytes=0, priority=50,ipv6,metadata=0x30d40/0xfffffe,ipv6_dst=2001:db8:1234:0:f816:3eff:fef5:5b21 actions=write_metad
ata:0x2/0xfffffe,goto_table:213
 cookie=0x6900000, duration=4215.377s, table=212, n_packets=3, n_bytes=270, priority=0 actions=goto_table:213
 cookie=0x6900000, duration=4215.377s, table=213, n_packets=6, n_bytes=588, priority=62020,ct_state=-new+est-rel-inv+trk actions=resubmit(,17)
 cookie=0x6900000, duration=4215.259s, table=213, n_packets=0, n_bytes=0, priority=62020,ct_state=-new-est+rel-inv+trk actions=resubmit(,17)
 cookie=0x6900000, duration=4215.377s, table=213, n_packets=0, n_bytes=0, priority=0 actions=drop
 cookie=0x8000230, duration=4215.502s, table=220, n_packets=313, n_bytes=16894, priority=6,reg6=0x200 actions=load:0x90000200->NXM_NX_REG6[],write_metadata:0/0xfffffffffe,goto_table
:230
 cookie=0x8000007, duration=4215.502s, table=220, n_packets=313, n_bytes=16894, priority=9,reg6=0x90000200 actions=output:1
 cookie=0x8000230, duration=4215.377s, table=230, n_packets=313, n_bytes=16894, priority=0 actions=resubmit(,220)
 cookie=0x8000231, duration=4215.377s, table=231, n_packets=0, n_bytes=0, priority=0 actions=resubmit(,220)
 cookie=0x6900000, duration=4214.738s, table=239, n_packets=4, n_bytes=392, priority=62020,ct_state=+trk,ip actions=ct(table=241)
 cookie=0x6900000, duration=4214.579s, table=239, n_packets=0, n_bytes=0, priority=62020,ct_state=+trk,ipv6 actions=ct(table=241)
 cookie=0x6900000, duration=4214.579s, table=239, n_packets=479, n_bytes=24742, priority=61010 actions=goto_table:241
 cookie=0x6900000, duration=4215.174s, table=241, n_packets=2, n_bytes=684, priority=61010,ip,dl_dst=ff:ff:ff:ff:ff:ff,nw_dst=255.255.255.255 actions=goto_table:242
 cookie=0x6900000, duration=4215.174s, table=241, n_packets=0, n_bytes=0, priority=61005,dl_dst=ff:ff:ff:ff:ff:ff actions=resubmit(,220)
 cookie=0x6900000, duration=4215.174s, table=241, n_packets=14, n_bytes=840, priority=0 actions=drop
 cookie=0x6900000, duration=3782.906s, table=242, n_packets=0, n_bytes=0, priority=50,ipv6,metadata=0x138a000000/0xffff000000,ipv6_src=fe80::f816:3eff:fe5f:68b9 actions=write_metada
ta:0x2/0xfffffe,goto_table:243
 cookie=0x6900000, duration=3782.727s, table=242, n_packets=0, n_bytes=0, priority=50,ipv6,metadata=0x138a000000/0xffff000000,ipv6_src=2001:db8:1234:0:f816:3eff:fe5f:68b9 actions=wr
ite_metadata:0x2/0xfffffe,goto_table:243
 cookie=0x6900000, duration=3742.917s, table=242, n_packets=0, n_bytes=0, priority=50,ipv6,metadata=0x138a000000/0xffff000000,ipv6_src=2001:db8:1234:0:f816:3eff:fefe:71f1 actions=wr
ite_metadata:0x2/0xfffffe,goto_table:243
 cookie=0x6900000, duration=3742.448s, table=242, n_packets=0, n_bytes=0, priority=50,ipv6,metadata=0x138a000000/0xffff000000,ipv6_src=fe80::f816:3eff:fefe:71f1 actions=write_metada
ta:0x2/0xfffffe,goto_table:243
 cookie=0x6900000, duration=3782.837s, table=242, n_packets=2, n_bytes=196, priority=50,ip,metadata=0x138a000000/0xffff000000,nw_src=10.1.1.11 actions=write_metadata:0x2/0xfffffe,go
to_table:243
 cookie=0x6900000, duration=3742.740s, table=242, n_packets=2, n_bytes=196, priority=50,ip,metadata=0x138a000000/0xffff000000,nw_src=10.1.1.14 actions=write_metadata:0x2/0xfffffe,go
to_table:243
 cookie=0x6900000, duration=3664.072s, table=242, n_packets=0, n_bytes=0, priority=50,ip,metadata=0x30d40/0xfffffe,nw_src=10.1.1.5 actions=write_metadata:0x2/0xfffffe,goto_table:243
 cookie=0x6900000, duration=3660.218s, table=242, n_packets=0, n_bytes=0, priority=50,ip,metadata=0x30d40/0xfffffe,nw_src=10.1.1.6 actions=write_metadata:0x2/0xfffffe,goto_table:243
 cookie=0x6900000, duration=3663.276s, table=242, n_packets=0, n_bytes=0, priority=50,ipv6,metadata=0x30d40/0xfffffe,ipv6_src=fe80::f816:3eff:fece:2cc5 actions=write_metadata:0x2/0x
fffffe,goto_table:243
 cookie=0x6900000, duration=3662.088s, table=242, n_packets=0, n_bytes=0, priority=50,ipv6,metadata=0x30d40/0xfffffe,ipv6_src=2001:db8:1234:0:f816:3eff:fece:2cc5 actions=write_metad
ata:0x2/0xfffffe,goto_table:243
 cookie=0x6900000, duration=3660.318s, table=242, n_packets=0, n_bytes=0, priority=50,ipv6,metadata=0x30d40/0xfffffe,ipv6_src=fe80::f816:3eff:fef5:5b21 actions=write_metadata:0x2/0x
fffffe,goto_table:243
 cookie=0x6900000, duration=3660.100s, table=242, n_packets=0, n_bytes=0, priority=50,ipv6,metadata=0x30d40/0xfffffe,ipv6_src=2001:db8:1234:0:f816:3eff:fef5:5b21 actions=write_metad
ata:0x2/0xfffffe,goto_table:243
 cookie=0x6900000, duration=4215.174s, table=242, n_packets=2, n_bytes=684, priority=0 actions=goto_table:243
 cookie=0x6900000, duration=4215.174s, table=243, n_packets=9, n_bytes=882, priority=62020,ct_state=-new+est-rel-inv+trk actions=resubmit(,220)
 cookie=0x6900000, duration=4215.117s, table=243, n_packets=0, n_bytes=0, priority=62020,ct_state=-new-est+rel-inv+trk actions=resubmit(,220)
 cookie=0x6900000, duration=4215.174s, table=243, n_packets=2, n_bytes=684, priority=0 actions=drop

 Comments   
Comment by Kiran Vasudeva [ 18/Jan/18 ]

Hi Rajan,

I could not see stale flows with the test case you 

Steps Followed:

  1. Create 2 networks N1, N2.
  2. Create sub networks SubN1, SubN2.
  3. Create Router R1, add router-interface 1 for each SubN1, SubN2.
  4. Create port p1, p2.
  5. Add 2 VM's VM1(p1), VM2(p2) with Default SG. (also tried with VM's having Custom-SG SG-1 with only ICMP rules)
  6. Ping test from both VM. 
  7. Delete VM's, router-interface, networks, router (in the same order mentioned)

I could see all the flows from 212 and 242 are getting removed.

Could you share exact steps with any Custom-SG rules which could result in stale flows?

Comment by RajaRajan Manickam [ 14/Feb/18 ]

Hi Kiran,

 

 Still issues are reproducible.

 

Thanks,

RajaRajan

Comment by Kiran Vasudeva [ 14/Feb/18 ]

Hi RajaRajan,

Could you share exact steps with any Custom-SG rules which could result in stale flows?

 

Thanks,

Kiran

Comment by RajaRajan Manickam [ 14/Feb/18 ]

Hi Kiran,

 

 Please find a easy way to reproduce.

 

  1. Create N1(DualStack)
  2. Create N2(IPv4) and N3(IPV4)
  3. Create L3VPN and Associate N2 and N3
  4. Associate Router to L3VPN
  5. Delete L3VPN
  6. Delete Router
  7. Delete all ports,Subnets and Networks.

Thanks,

RajaRajan

Comment by Kiran Vasudeva [ 22/Feb/18 ]

Hi RajaRajan,

I tried the steps you mentioned with default SG, 2 times on last Wednesday. apparently the flows in 212/242 got deleted, the moment VM's are deleted. but could not update the Jira task. 

could you retry and share

  • exact steps.
  • any specific SG rules configured on the vm's whose stale flows are retained

 

Thanks,

Kiran

Comment by RajaRajan Manickam [ 19/Mar/18 ]

Issue is still reproducible with Dualstack VM.

 

Steps:

  1. Create N1 with S1(ipv4) and S2(ipv6)
  2. create router and associate S2 with it.
  3. Boot all VMs and check flows
  4. Create L3VPN and associate router to it.
  5. Delete L3VPN
  6. Remove subnet from router
  7. delete all VM's, Ports,Subnet and Network
  8. Check dump-flows
Comment by Kiran Vasudeva [ 20/Mar/18 ]

Hi Rajarajan,

I tried the steps you have shared, and could see all sg rules related to vm/port are deleted. Also since the Acl-pipeline has changed in master/oxygen builds, the ip address of vm would not be in 212/242 tables.

could you share build details, also below logs, before DELETE L3vpn/VM/Port is executed:

  1. nova list
  2. neutron net-list
  3. neutron port-list 
  4. neutron port-show <port-id>
  5. neutron security-group-list
  6. neutron security-group-show <sg-id>
  7. sudo ovs-vsctl show
  8. sudo ovs-ofctl -OOpenFlow13 dump-flows br-int 
  9. neutron router-show <router-id>
  10. neutron bgpvpn-show <bgpvpn-id>

and after DELETEoperation:

  1. sudo ovs-ofctl -OOpenFlow13 dump-flows br-int 
  2. neutron port-list

 

-Kiran

Generated at Wed Feb 07 20:23:07 UTC 2024 using Jira 8.20.10#820010-sha1:ace47f9899e9ee25d7157d59aa17ab06aee30d3d.