[NETVIRT-581] Tables/Groups do not get cleanup when VM/ports/networks are deleted Created: 31/Mar/17  Updated: 08/Nov/19  Resolved: 08/Nov/19

Status: Resolved
Project: netvirt
Component/s: General
Affects Version/s: Boron
Fix Version/s: None

Type: Bug Priority: Medium
Reporter: Nikolas Hermanns Assignee: Kiran Upadhyaya
Resolution: Won't Do Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Operating System: All
Platform: All

Attachments: Text File stale-flow-group-odl-ds-dumps.txt    
External issue ID: 8132

 Description   

In ovs I can see a lot of remaining tables/groups although all stuff is deleted from openstack. The only remainin stuff should be the vxlan tunnels or? If tables are not cleanup that can last in difficulties when wrong matching happens later! One thing I noticed it that the vpn-interfaces and the vpn-instances are not deleted at all in the data store.

OFPST_FLOW reply (OF1.3) (xid=0x2):
table=0, n_packets=14528, n_bytes=1068589, priority=5,in_port=3 actions=write_metadata:0x30000000001/0xfffff0000000001,goto_table:36
table=0, n_packets=13118, n_bytes=994518, priority=1,in_port=1 actions=write_metadata:0xb0000000001/0xffffff0000000001,goto_table:17
table=0, n_packets=3434, n_bytes=269557, priority=5,in_port=5 actions=write_metadata:0x140000000001/0xfffff0000000001,goto_table:36
table=0, n_packets=4604, n_bytes=381230, priority=5,in_port=7 actions=write_metadata:0x190000000001/0xfffff0000000001,goto_table:36
table=17, n_packets=13117, n_bytes=994464, priority=5,metadata=0xb0000000000/0xffffff0000000000 actions=write_metadata:0x60000b00000222e0/0xfffffffffffffffe,goto_table:19
table=17, n_packets=13106, n_bytes=993347, priority=6,metadata=0x60000b0000000000/0xffffff0000000000 actions=load:0xb->NXM_NX_REG1[0..19],write_metadata:0x70000b138b000000/0xfffffffffffffffe,goto_table:48
table=17, n_packets=792, n_bytes=76738, priority=5,metadata=0x5000120000000000/0xffffff0000000000 actions=write_metadata:0x60001200000222ec/0xfffffffffffffffe,goto_table:19
table=17, n_packets=667, n_bytes=62902, priority=5,metadata=0x5000160000000000/0xffffff0000000000 actions=write_metadata:0x6000160000022308/0xfffffffffffffffe,goto_table:19
table=17, n_packets=0, n_bytes=0, priority=0,metadata=0x5000000000000000/0xf000000000000000 actions=write_metadata:0x6000000000000000/0xf000000000000000,goto_table:80
table=18, n_packets=0, n_bytes=0, priority=0 actions=goto_table:38
table=19, n_packets=87, n_bytes=3654, priority=100,arp,arp_op=2 actions=CONTROLLER:65535,resubmit(,17)
table=19, n_packets=211, n_bytes=8862, priority=100,arp,arp_op=1 actions=group:5000
table=19, n_packets=16043, n_bytes=1280518, priority=0 actions=resubmit(,17)
table=20, n_packets=0, n_bytes=0, priority=0 actions=goto_table:80
table=21, n_packets=1, n_bytes=98, priority=42,icmp,metadata=0x222ea/0xfffffffe,nw_dst=10.10.10.1,icmp_type=8,icmp_code=0 actions=move:NXM_OF_ETH_SRC[]>NXM_OF_ETH_DST[],set_field:fa:16:3e:eb:b3:02>eth_src,move:NXM_OF_IP_SRC[]>NXM_OF_IP_DST[],set_field:10.10.10.1>ip_src,set_field:0->icmp_type,load:0->NXM_OF_IN_PORT[],resubmit(,21)
table=22, n_packets=0, n_bytes=0, priority=0 actions=CONTROLLER:65535
table=36, n_packets=0, n_bytes=0, priority=5,tun_id=0x138b actions=write_actions(group:210005)
table=36, n_packets=0, n_bytes=0, priority=5,tun_id=0xb actions=load:0xb00->NXM_NX_REG6[],resubmit(,220)
table=40, n_packets=9, n_bytes=810, priority=0 actions=drop
table=41, n_packets=1576, n_bytes=151597, priority=62020,ct_state=-new+est-rel-inv+trk actions=resubmit(,17)
table=41, n_packets=0, n_bytes=0, priority=62020,ct_state=-new-est+rel-inv+trk actions=resubmit(,17)
table=41, n_packets=0, n_bytes=0, priority=62020,ct_state=+inv+trk actions=drop
table=41, n_packets=0, n_bytes=0, priority=50,ct_state=+new+trk actions=drop
table=41, n_packets=0, n_bytes=0, priority=0 actions=drop
table=45, n_packets=0, n_bytes=0, priority=0 actions=resubmit(,17)
table=48, n_packets=16499, n_bytes=1307268, priority=0 actions=resubmit(,49),resubmit(,50)
table=50, n_packets=8167, n_bytes=441506, idle_timeout=300, send_flow_rem priority=20,metadata=0xb138b000000/0xfffffffff000000,dl_src=52:54:00:0b:9a:a1 actions=goto_table:51
table=50, n_packets=1628, n_bytes=183964, idle_timeout=300, send_flow_rem priority=20,metadata=0xb138b000000/0xfffffffff000000,dl_src=ce:f7:d9:be:5c:cc actions=goto_table:51
table=50, n_packets=1628, n_bytes=180708, idle_timeout=300, send_flow_rem priority=20,metadata=0xb138b000000/0xfffffffff000000,dl_src=6e:67:ff:8c:e4:1b actions=goto_table:51
table=50, n_packets=1628, n_bytes=183964, idle_timeout=300, send_flow_rem priority=20,metadata=0xb138b000000/0xfffffffff000000,dl_src=42:1d:d5:75:fc:f9 actions=goto_table:51
table=50, n_packets=6, n_bytes=523, priority=10,reg4=0x1 actions=goto_table:51
table=50, n_packets=36, n_bytes=1596, priority=0 actions=CONTROLLER:65535,learn(table=49,hard_timeout=10,priority=0,cookie=0x8600000,NXM_OF_ETH_SRC[],NXM_NX_REG1[0..19],load:0x1->NXM_NX_REG4[0..7]),goto_table:51
table=51, n_packets=0, n_bytes=0, priority=20,metadata=0x138b000000/0xffff000000,dl_dst=52:54:00:0b:9a:a1 actions=load:0xb00->NXM_NX_REG6[],resubmit(,220)
table=51, n_packets=0, n_bytes=0, priority=20,metadata=0x138b000000/0xffff000000,dl_dst=ce:f7:d9:be:5c:cc actions=load:0xb00->NXM_NX_REG6[],resubmit(,220)
table=51, n_packets=0, n_bytes=0, priority=20,metadata=0x138b000000/0xffff000000,dl_dst=6e:67:ff:8c:e4:1b actions=load:0xb00->NXM_NX_REG6[],resubmit(,220)
table=51, n_packets=0, n_bytes=0, priority=20,metadata=0x138b000000/0xffff000000,dl_dst=42:1d:d5:75:fc:f9 actions=load:0xb00->NXM_NX_REG6[],resubmit(,220)
table=51, n_packets=13241, n_bytes=1004259, priority=0 actions=goto_table:52
table=52, n_packets=13106, n_bytes=993347, priority=5,metadata=0x138b000001/0xffff000001 actions=write_actions(group:210005)
table=52, n_packets=0, n_bytes=0, priority=5,metadata=0x138b000000/0xffff000001 actions=write_actions(group:210006)
table=55, n_packets=0, n_bytes=0, priority=10,tun_id=0xb,metadata=0xb0000000000/0xfffff0000000000 actions=drop
table=55, n_packets=0, n_bytes=0, priority=9,tun_id=0xb actions=load:0xb00->NXM_NX_REG6[],resubmit(,220)
table=60, n_packets=0, n_bytes=0, priority=0 actions=resubmit(,17)
table=80, n_packets=0, n_bytes=0, priority=0 actions=resubmit(,17)
table=81, n_packets=20, n_bytes=840, priority=100,arp,metadata=0x60001200000222ec/0xffffff00fffffffe,arp_tpa=10.10.10.1,arp_op=1 actions=move:NXM_OF_ETH_SRC[]>NXM_OF_ETH_DST[],set_field:fa:16:3e:f8:3d:d6>eth_src,load:0x2->NXM_OF_ARP_OP[],move:NXM_NX_ARP_SHA[]>NXM_NX_ARP_THA[],move:NXM_OF_ARP_SPA[]>NXM_OF_ARP_TPA[],load:0xfa163ef83dd6->NXM_NX_ARP_SHA[],load:0xa0a0a01->NXM_OF_ARP_SPA[],load:0->NXM_OF_IN_PORT[],load:0x1200->NXM_NX_REG6[],resubmit(,220)
table=81, n_packets=0, n_bytes=0, priority=100,arp,metadata=0x6000160000022308/0xffffff00fffffffe,arp_tpa=10.10.10.1,arp_op=1 actions=move:NXM_OF_ETH_SRC[]>NXM_OF_ETH_DST[],set_field:fa:16:3e:eb:b3:02>eth_src,load:0x2->NXM_OF_ARP_OP[],move:NXM_NX_ARP_SHA[]>NXM_NX_ARP_THA[],move:NXM_OF_ARP_SPA[]>NXM_OF_ARP_TPA[],load:0xfa163eebb302->NXM_NX_ARP_SHA[],load:0xa0a0a01->NXM_OF_ARP_SPA[],load:0->NXM_OF_IN_PORT[],load:0x1600->NXM_NX_REG6[],resubmit(,220)
table=81, n_packets=137, n_bytes=5754, priority=0 actions=drop
table=220, n_packets=0, n_bytes=0, priority=8,reg6=0xb00,metadata=0x1/0x1 actions=drop
table=220, n_packets=304, n_bytes=25232, priority=7,reg6=0xb00 actions=output:1
table=220, n_packets=787, n_bytes=76434, priority=7,reg6=0x70001200 actions=output:4
table=220, n_packets=763, n_bytes=71540, priority=7,reg6=0x70001600 actions=output:6
table=251, n_packets=16, n_bytes=2260, priority=0 actions=drop
table=252, n_packets=4394, n_bytes=426033, priority=62020,ct_state=-new+est-rel-inv+trk actions=resubmit(,220)
table=252, n_packets=0, n_bytes=0, priority=62020,ct_state=-new-est+rel-inv+trk actions=resubmit(,220)
table=252, n_packets=3, n_bytes=378, priority=62020,ct_state=+inv+trk actions=drop
table=252, n_packets=0, n_bytes=0, priority=50,ct_state=+new+trk actions=drop
table=252, n_packets=0, n_bytes=0, priority=0 actions=drop
OFPST_GROUP_DESC reply (OF1.3) (xid=0x2):
group_id=210006,type=all,bucket=actions=group:210005,bucket=actions=set_field:0x138b->tun_id,output:3,bucket=actions=load:0xb00->NXM_NX_REG6[],resubmit(,220)
group_id=150005,type=all,bucket=actions=set_field:fa:16:3e:f8:3d:d6->eth_src,set_field:fa:16:3e:c1:e7:f3->eth_dst,load:0x1200->NXM_NX_REG6[],resubmit(,220)
group_id=200000,type=all,bucket=actions=set_field:00:72:2b:16:87:6c->eth_dst,load:0xb00->NXM_NX_REG6[],resubmit(,220)
group_id=210005,type=all
group_id=5000,type=all,bucket=actions=CONTROLLER:65535,bucket=actions=resubmit(,17),bucket=actions=resubmit(,81)
1(br-ex-patch): addr:ea:a7:8e:97:12:2c
3(tunfcf91e373ca): addr:a2:5a:37:6f:f3:4a
5(tunbc3fdd30422): addr:d6:fa:df:c2:f7:c6
7(tunffa5e6e85eb): addr:8a:5b:95:8d:9b:77
LOCAL(br-int): addr:b0:51:0d:b9:ce:b0

in Openstack:

  1. neutron port-list

nova list
-------------------------------------------+

ID Name Status Task State Power State Networks

-------------------------------------------+
-------------------------------------------+

In datastore:
>>> odl.o()
Sending METHOD (get) URL (http://192.0.2.4:8081/controller/nb/v2/neutron/networks) JSON (None)
Sending METHOD (get) URL (http://192.0.2.4:8081/controller/nb/v2/neutron/networks) JSON (None)
Sending METHOD (get) URL (http://192.0.2.4:8081/controller/nb/v2/neutron/ports) JSON (None)
------------Networks----------------
[

{ u'admin_state_up': True, u'id': u'd650c111-ef3f-41db-a7d4-c46b1617a670', u'name': u'external', u'provider:network_type': u'flat', u'provider:physical_network': u'datacentre', u'router:external': True, u'segments': [], u'shared': False, u'status': u'ACTIVE', u'tenant_id': u'5eccb3654b994f5faaf1cbc53966adf9'}

]
------------Subnets----------------
[

{ u'admin_state_up': True, u'id': u'd650c111-ef3f-41db-a7d4-c46b1617a670', u'name': u'external', u'provider:network_type': u'flat', u'provider:physical_network': u'datacentre', u'router:external': True, u'segments': [], u'shared': False, u'status': u'ACTIVE', u'tenant_id': u'5eccb3654b994f5faaf1cbc53966adf9'}

]
------------Ports----------------
[]

------------l3_vpn_instance----------------
Sending METHOD (get) URL (http://192.0.2.4:8081/restconf/config/l3vpn:vpn-instances) JSON (None)
{ u'vpn-instances': { u'vpn-instance': [ { u'ipv4-family': { u'route-distinguisher': u'12:12',
u'vpnTargets': { u'vpnTarget': [

{ u'vrfRTType': u'export_extcommunity', u'vrfRTValue': u'55:55'}

,

{ u'vrfRTType': u'import_extcommunity', u'vrfRTValue': u'88:88'}

]}},
u'vpn-instance-name': u'8dbb1406-9dc8-4972-88f9-d56270f55a43'},
{ u'ipv4-family': { u'vpnTargets': { }},
u'vpn-instance-name': u'd650c111-ef3f-41db-a7d4-c46b1617a670'},
{ u'ipv4-family': { u'route-distinguisher': u'18:18',
u'vpnTargets': { u'vpnTarget': [

{ u'vrfRTType': u'both', u'vrfRTValue': u'88:88'}

]}},
u'vpn-instance-name': u'a493adf2-2a47-4c04-9540-54a754cdc9f3'},
{ u'ipv4-family': { u'route-distinguisher': u'18:18',
u'vpnTargets': { u'vpnTarget': [

{ u'vrfRTType': u'both', u'vrfRTValue': u'88:88'}

]}},
u'vpn-instance-name': u'6edc0fc0-f3ad-4df7-ac33-971949b962ea'}]}}
------------vpn_interfaces----------------
Sending METHOD (get) URL (http://192.0.2.4:8081/restconf/config/l3vpn:vpn-interfaces) JSON (None)
{ u'vpn-interfaces': { u'vpn-interface': [ { u'is-router-interface': False,
u'name': u'a03f221a-05f0-4df6-b03b-9de34dc25d9e',
u'odl-l3vpn:adjacency': [

{ u'ip_address': u'10.10.10.13/32', u'mac_address': u'fa:16:3e:59:3e:46', u'primary-adjacency': True, u'subnet_id': u'144421c5-2654-462d-bcdd-0bb0adc13a95'}

],
u'vpn-instance-name': u'8dbb1406-9dc8-4972-88f9-d56270f55a43'},

{ u'is-router-interface': False, u'name': u'145264332241977:br-ex-patch:flat', u'vpn-instance-name': u'd650c111-ef3f-41db-a7d4-c46b1617a670'}

,

{ u'is-router-interface': False, u'name': u'53407969530085:br-ex-patch:flat', u'vpn-instance-name': u'd650c111-ef3f-41db-a7d4-c46b1617a670'}

,
{ u'is-router-interface': False,
u'name': u'20ebd92d-f0ea-40aa-9f67-b6c52305ba53',
u'odl-l3vpn:adjacency': [

{ u'ip_address': u'10.10.11.2/32', u'mac_address': u'fa:16:3e:a9:b1:ce', u'primary-adjacency': True, u'subnet_id': u'8388edb0-ccdf-4a1c-b7a7-eaf0d10709c2'}

],
u'vpn-instance-name': u'99a4fe5c-5f6d-4765-b7c1-f6c6007c1a3d'},
{ u'is-router-interface': False,
u'name': u'2c446de6-be84-4622-8004-818475b49c17',
u'odl-l3vpn:adjacency': [

{ u'ip_address': u'10.10.10.2/32', u'mac_address': u'fa:16:3e:61:99:9d', u'primary-adjacency': True, u'subnet_id': u'5a939ff4-4a35-47e5-bf46-ffea9f2c49df'}

],
u'vpn-instance-name': u'dd88fdb4-cfdc-4ed4-a0ac-d75d15ab5f77'},
{ u'is-router-interface': False,
u'name': u'14eb672d-2c8d-4f5e-b68e-46633bf3dc01',
u'odl-l3vpn:adjacency': [

{ u'ip_address': u'10.10.11.5/32', u'mac_address': u'fa:16:3e:0b:ac:bf', u'primary-adjacency': True, u'subnet_id': u'8388edb0-ccdf-4a1c-b7a7-eaf0d10709c2'}

],
u'vpn-instance-name': u'99a4fe5c-5f6d-4765-b7c1-f6c6007c1a3d'},
{ u'is-router-interface': False,
u'name': u'2ea9bc81-9aa9-4aef-baad-7ed0b09d4e9b',
u'odl-l3vpn:adjacency': [

{ u'ip_address': u'10.10.10.12/32', u'mac_address': u'fa:16:3e:c1:e7:f3', u'primary-adjacency': True, u'subnet_id': u'96d2649e-924b-4e08-8889-be28c2df9883'}

],
u'vpn-instance-name': u'a493adf2-2a47-4c04-9540-54a754cdc9f3'},
{ u'is-router-interface': False,
u'name': u'6db295d9-48af-4334-abcb-119f70445323',
u'odl-l3vpn:adjacency': [

{ u'ip_address': u'10.10.11.2/32', u'mac_address': u'fa:16:3e:55:84:db', u'primary-adjacency': True, u'subnet_id': u'e53a5af0-0ee3-4704-b72a-9474ecef3463'}

],
u'vpn-instance-name': u'8dbb1406-9dc8-4972-88f9-d56270f55a43'},
{ u'is-router-interface': False,
u'name': u'49243102-f92b-4289-b570-b7acc0c211c1',
u'odl-l3vpn:adjacency': [

{ u'ip_address': u'10.10.10.14/32', u'mac_address': u'fa:16:3e:fc:81:34', u'primary-adjacency': True, u'subnet_id': u'144421c5-2654-462d-bcdd-0bb0adc13a95'}

],
u'vpn-instance-name': u'8dbb1406-9dc8-4972-88f9-d56270f55a43'},
{ u'is-router-interface': False,
u'name': u'e9448960-b7e7-4746-bac7-2eb7979e03cc',
u'odl-l3vpn:adjacency': [

{ u'ip_address': u'10.10.10.2/32', u'mac_address': u'fa:16:3e:a1:7c:b3', u'primary-adjacency': True, u'subnet_id': u'144421c5-2654-462d-bcdd-0bb0adc13a95'}

],
u'vpn-instance-name': u'8dbb1406-9dc8-4972-88f9-d56270f55a43'},

{ u'is-router-interface': False, u'name': u'193862169120432:br-ex-patch:flat', u'vpn-instance-name': u'd650c111-ef3f-41db-a7d4-c46b1617a670'}

,
{ u'is-router-interface': False,
u'name': u'9d420291-6d4e-434f-abab-c5aa0136f5cd',
u'odl-l3vpn:adjacency': [

{ u'ip_address': u'10.10.10.3/32', u'mac_address': u'fa:16:3e:c5:3f:60', u'primary-adjacency': True, u'subnet_id': u'144421c5-2654-462d-bcdd-0bb0adc13a95'}

],
u'vpn-instance-name': u'8dbb1406-9dc8-4972-88f9-d56270f55a43'},

{ u'is-router-interface': False, u'name': u'119119107917516:br-ex-patch:flat', u'vpn-instance-name': u'd650c111-ef3f-41db-a7d4-c46b1617a670'}

,
{ u'is-router-interface': False,
u'name': u'40343197-9e38-4263-9b24-71d527d3566e',
u'odl-l3vpn:adjacency': [

{ u'ip_address': u'10.10.10.2/32', u'mac_address': u'fa:16:3e:e6:de:c5', u'primary-adjacency': True, u'subnet_id': u'96d2649e-924b-4e08-8889-be28c2df9883'}

],
u'vpn-instance-name': u'a493adf2-2a47-4c04-9540-54a754cdc9f3'}]}}
------------ietf_interface----------------
Sending METHOD (get) URL (http://192.0.2.4:8081/restconf/config/ietf-interfaces:interfaces) JSON (None)
{ u'interfaces': { u'interface': [

{ u'description': u'VXLAN Trunk Interface', u'enabled': True, u'name': u'tun9148d1ea761', u'odl-interface:datapath-node-identifier': 53407969530085, u'odl-interface:internal': True, u'odl-interface:monitor-enabled': True, u'odl-interface:monitor-interval': 1000, u'odl-interface:monitor-protocol': u'odl-interface:tunnel-monitoring-type-bfd', u'odl-interface:tunnel-destination': u'11.0.0.22', u'odl-interface:tunnel-gateway': u'0.0.0.0', u'odl-interface:tunnel-interface-type': u'odl-interface:tunnel-type-vxlan', u'odl-interface:tunnel-source': u'11.0.0.30', u'type': u'iana-if-type:tunnel'}

,

{ u'description': u'VXLAN Trunk Interface', u'enabled': True, u'name': u'tunbc3fdd30422', u'odl-interface:datapath-node-identifier': 193862169120432, u'odl-interface:internal': True, u'odl-interface:monitor-enabled': True, u'odl-interface:monitor-interval': 1000, u'odl-interface:monitor-protocol': u'odl-interface:tunnel-monitoring-type-bfd', u'odl-interface:tunnel-destination': u'11.0.0.27', u'odl-interface:tunnel-gateway': u'0.0.0.0', u'odl-interface:tunnel-interface-type': u'odl-interface:tunnel-type-vxlan', u'odl-interface:tunnel-source': u'11.0.0.22', u'type': u'iana-if-type:tunnel'}

,

{ u'enabled': True, u'name': u'53407969530085:br-ex-patch:flat', u'odl-interface:external': True, u'odl-interface:l2vlan-mode': u'transparent', u'odl-interface:parent-interface': u'53407969530085:br-ex-patch', u'type': u'iana-if-type:l2vlan'}

,

{ u'description': u'VXLAN Trunk Interface', u'enabled': True, u'name': u'tunfcf91e373ca', u'odl-interface:datapath-node-identifier': 193862169120432, u'odl-interface:internal': True, u'odl-interface:monitor-enabled': True, u'odl-interface:monitor-interval': 1000, u'odl-interface:monitor-protocol': u'odl-interface:tunnel-monitoring-type-bfd', u'odl-interface:tunnel-destination': u'11.0.0.30', u'odl-interface:tunnel-gateway': u'0.0.0.0', u'odl-interface:tunnel-interface-type': u'odl-interface:tunnel-type-vxlan', u'odl-interface:tunnel-source': u'11.0.0.22', u'type': u'iana-if-type:tunnel'}

,

{ u'description': u'VXLAN Trunk Interface', u'enabled': True, u'name': u'tun15710567189', u'odl-interface:datapath-node-identifier': 53407969530085, u'odl-interface:internal': True, u'odl-interface:monitor-enabled': True, u'odl-interface:monitor-interval': 1000, u'odl-interface:monitor-protocol': u'odl-interface:tunnel-monitoring-type-bfd', u'odl-interface:tunnel-destination': u'11.0.0.27', u'odl-interface:tunnel-gateway': u'0.0.0.0', u'odl-interface:tunnel-interface-type': u'odl-interface:tunnel-type-vxlan', u'odl-interface:tunnel-source': u'11.0.0.30', u'type': u'iana-if-type:tunnel'}

,

{ u'description': u'VXLAN Trunk Interface', u'enabled': True, u'name': u'tun60867c0db22', u'odl-interface:datapath-node-identifier': 119119107917516, u'odl-interface:internal': True, u'odl-interface:monitor-enabled': True, u'odl-interface:monitor-interval': 1000, u'odl-interface:monitor-protocol': u'odl-interface:tunnel-monitoring-type-bfd', u'odl-interface:tunnel-destination': u'11.0.0.22', u'odl-interface:tunnel-gateway': u'0.0.0.0', u'odl-interface:tunnel-interface-type': u'odl-interface:tunnel-type-vxlan', u'odl-interface:tunnel-source': u'11.0.0.27', u'type': u'iana-if-type:tunnel'}

,

{ u'description': u'VXLAN Trunk Interface', u'enabled': True, u'name': u'tun0733a8cf82e', u'odl-interface:datapath-node-identifier': 145264332241977, u'odl-interface:internal': True, u'odl-interface:monitor-enabled': True, u'odl-interface:monitor-interval': 1000, u'odl-interface:monitor-protocol': u'odl-interface:tunnel-monitoring-type-bfd', u'odl-interface:tunnel-destination': u'11.0.0.22', u'odl-interface:tunnel-gateway': u'0.0.0.0', u'odl-interface:tunnel-interface-type': u'odl-interface:tunnel-type-vxlan', u'odl-interface:tunnel-source': u'11.0.0.24', u'type': u'iana-if-type:tunnel'}

,

{ u'enabled': True, u'name': u'145264332241977:br-ex-patch:flat', u'odl-interface:external': True, u'odl-interface:l2vlan-mode': u'transparent', u'odl-interface:parent-interface': u'145264332241977:br-ex-patch', u'type': u'iana-if-type:l2vlan'}

,

{ u'enabled': True, u'name': u'119119107917516:br-ex-patch:flat', u'odl-interface:external': True, u'odl-interface:l2vlan-mode': u'transparent', u'odl-interface:parent-interface': u'119119107917516:br-ex-patch', u'type': u'iana-if-type:l2vlan'}

,

{ u'enabled': True, u'name': u'193862169120432:br-ex-patch:flat', u'odl-interface:external': True, u'odl-interface:l2vlan-mode': u'transparent', u'odl-interface:parent-interface': u'193862169120432:br-ex-patch', u'type': u'iana-if-type:l2vlan'}

,

{ u'description': u'VXLAN Trunk Interface', u'enabled': True, u'name': u'tunffa5e6e85eb', u'odl-interface:datapath-node-identifier': 193862169120432, u'odl-interface:internal': True, u'odl-interface:monitor-enabled': True, u'odl-interface:monitor-interval': 1000, u'odl-interface:monitor-protocol': u'odl-interface:tunnel-monitoring-type-bfd', u'odl-interface:tunnel-destination': u'11.0.0.24', u'odl-interface:tunnel-gateway': u'0.0.0.0', u'odl-interface:tunnel-interface-type': u'odl-interface:tunnel-type-vxlan', u'odl-interface:tunnel-source': u'11.0.0.22', u'type': u'iana-if-type:tunnel'}

,

{ u'description': u'VXLAN Trunk Interface', u'enabled': True, u'name': u'tun3e4a3318e78', u'odl-interface:datapath-node-identifier': 145264332241977, u'odl-interface:internal': True, u'odl-interface:monitor-enabled': True, u'odl-interface:monitor-interval': 1000, u'odl-interface:monitor-protocol': u'odl-interface:tunnel-monitoring-type-bfd', u'odl-interface:tunnel-destination': u'11.0.0.30', u'odl-interface:tunnel-gateway': u'0.0.0.0', u'odl-interface:tunnel-interface-type': u'odl-interface:tunnel-type-vxlan', u'odl-interface:tunnel-source': u'11.0.0.24', u'type': u'iana-if-type:tunnel'}

,

{ u'description': u'VXLAN Trunk Interface', u'enabled': True, u'name': u'tun10430618980', u'odl-interface:datapath-node-identifier': 119119107917516, u'odl-interface:internal': True, u'odl-interface:monitor-enabled': True, u'odl-interface:monitor-interval': 1000, u'odl-interface:monitor-protocol': u'odl-interface:tunnel-monitoring-type-bfd', u'odl-interface:tunnel-destination': u'11.0.0.30', u'odl-interface:tunnel-gateway': u'0.0.0.0', u'odl-interface:tunnel-interface-type': u'odl-interface:tunnel-type-vxlan', u'odl-interface:tunnel-source': u'11.0.0.27', u'type': u'iana-if-type:tunnel'}

,
{ u'description': u'VXLAN Trunk Interface',
u'enabled': True,
u'name': u'tun1d81037158f',
u'odl-interface:datapath-node-identifier': 53407969530085,
u'odl-interface:internal': True,
u'odl-interface:monitor-enabled': True,
u'odl-interface:monitor-interval': 1000,
u'odl-interface:monitor-protocol': u'odl-interface:tunnel-monitoring-type-bfd',
u'odl-interface:tunnel-destination': u'11.0.0.24',
u'od...

 Comments   
Comment by Periyasamy Palanisamy [ 05/Apr/17 ]

The extra flow/group entries that you have here in ELAN is for flat network that you created (d650c111-ef3f-41db-a7d4-c46b1617a670). As long vlan/flat provider network created in ODL and when DPN is connected, then transparent interface is created (pointing to the patch-port attached to the physnet port) and elan service is bound to it. This will removed only when neutron-network is deleted (or) DPN is disconnected.

The ELAN remote BC group is having buckets for both vxlan tunnel and flat network's transparent port.

group_id=210006,type=all,bucket=actions=group:210005,bucket=actions=set_field:0x138b->tun_id,output:3,bucket=actions=load:0xb00->NXM_NX_REG6[],resubmit(,220)

This is fixed in master with https://git.opendaylight.org/gerrit/#/c/54240/

For VPN related flow/group leftovers, I suggest Vivek to comment.

Comment by Vivekanandan Narasimhan [ 05/Apr/17 ]

[‎05-‎Apr-‎17 6:37 PM]
please give complete flow-dumps, rest-api outputs of config/vpn-instances, config/vpn-interfaces, then karaf logs
[‎05-‎Apr-‎17 6:38 PM]
and if possible , plesae collect all REST APIs available here:
[‎05-‎Apr-‎17 6:38 PM]
https://jenkins.opendaylight.org/releng/job/netvirt-csit-1node-openstack-newton-nodl-v2-gate-stateful-carbon/744/robot/report/log.html#s1-s2-s1-t11-k3-k1-k2-k1-k2

Comment by Nikolas Hermanns [ 05/Apr/17 ]

I see flows with mac adresses which are not running in the system anymore:
table=51, n_packets=0, n_bytes=0, priority=20,metadata=0x1389000000/0xffff000000,dl_dst=fa:16:3e:26:a2:4b actions=load:0x300->NXM_NX_REG6[],resubmit(,220)

in addition I see a residual group which breaks dhcp for every vm which connects.

group_id=5000,type=all,bucket=actions=CONTROLLER:65535,bucket=actions=resubmit(,17),bucket=actions=resubmit(,81)
group_id=210002,type=all,bucket=actions=group:210001,bucket=actions=load:0x300->NXM_NX_REG6[],resubmit(,220)
group_id=210001,type=all
group_id=150010,type=all,bucket=actions=set_field:fa:16:3e:33:a2:cc->eth_src,set_field:fa:16:3e:51:eb:2d->eth_dst,load:0x1000->NXM_NX_REG6[],resubmit(,220)
group_id=150003,type=all,bucket=actions=set_field:fe:16:3e:0c:92:f0->eth_src,set_field:fa:16:3e:0c:92:f0->eth_dst,load:0x500->NXM_NX_REG6[],resubmit(,220)

!!!Unable to render embedded object: File (group_id=210001) not found.!!!

here you find the whole datastore dump:
https://drive.google.com/open?id=0B_Rr7XjF0yoHLTZNSm40SDRzYkU

the node was completely clean:
[root@overcloud-controller-0 ~]# nova list
-------------------------------------------+

ID Name Status Task State Power State Networks

-------------------------------------------+
-------------------------------------------+
[root@overcloud-controller-0 ~]# neutron net-list

[root@overcloud-controller-0 ~]# neutron bgpvpn-list

[root@overcloud-controller-0 ~]# neutron port-list

All vms which where started before where started to 100% ACTIVE in openstack. However lots of them did not get an IP.

Comment by Kiran N Upadhyaya [ 10/Apr/17 ]

Hi Nikolas,

From the new Dumps, I was able to figure out that vpn-interface 72702c41-0d90-4eca-aa9e-26279737192a is stale. However, I could not figure out the exact cause. I see that a south-bound remove event has been triggered for this interface, but even that has not removed the table-21 flows.
I would need TRACE logs to further analyze this issue.
Could you please enable TRACE for logging in karaf and reproduce the issue again?
The karaf cli command to set TRACE logging level is:
log:set TRACE org.opendaylight.netvirt

Once reproduced, please collect all the datastore dumps, karaf logs and openflow groups/flows.

Thanks,
Kiran

Comment by Periyasamy Palanisamy [ 02/Jan/18 ]

In carbon, There are still stale entries present in table 21 (looks like ICMP responder flows for GW), VM nexthop groups and ACL flows in table 212 and 242 even after deleting VMs and its associated networks, routers and bgpvpns (attached the flow/group dumps from a compute and ds dumps from ODL).

Comment by Abhinav Gupta [ 08/Nov/19 ]

Old bug, will reopen if seen again

Generated at Wed Feb 07 20:21:56 UTC 2024 using Jira 8.20.10#820010-sha1:ace47f9899e9ee25d7157d59aa17ab06aee30d3d.