Multiple VM instance deletion doesnt remove the flow table entries including table=0

Deleting Multiple VM instances in a single nova cmd results in not all flows of a particular VM instance not getting deleted

Steps to reproduce the issue:

1. Spawn Multiple VMs (Ipv6) associated with security groups.
2. Verify the flows
3. Now deleted Multiple instances in single nova cmd.
4. OPenstack sends "delete' instance to ODL
5. Neutron port shows no VMs/ports in Neutron/port restconf
6. But table=0 and other entries of a particular VM instance is not deleted.

openstack commands
--------------------

neutron security-group-create SG-1
neutron security-group-create SG-11
neutron security-group-create SG-111

neutron security-group-rule-create --direction egress --protocol udp --port_range_min 10000 --port_range_max 10500 --ethertype IPv6 SG-1
neutron security-group-rule-create --direction ingress --protocol udp --port_range_min 10000 --port_range_max 10500 --ethertype IPv6 SG-1

neutron security-group-rule-create --direction egress --protocol tcp --port_range_min 17000 --port_range_max 17500 --ethertype IPv6 SG-1
neutron security-group-rule-create --direction ingress --protocol tcp --port_range_min 17000 --port_range_max 17500 --ethertype IPv6 SG-1

neutron security-group-rule-create --protocol icmpv6 --direction ingress --ethertype IPv6 SG-1
neutron security-group-rule-create --protocol icmpv6 --direction egress --ethertype IPv6 SG-1

neutron security-group-rule-create --direction egress --protocol udp --port_range_min 20000 --port_range_max 20500 --ethertype IPv6 SG-11
neutron security-group-rule-create --direction ingress --protocol udp --port_range_min 20000 --port_range_max 20500 --ethertype IPv6 SG-11

neutron security-group-rule-create --direction egress --protocol tcp --port_range_min 27000 --port_range_max 27500 --ethertype IPv6 SG-11
neutron security-group-rule-create --direction ingress --protocol tcp --port_range_min 27000 --port_range_max 27500 --ethertype IPv6 SG-11

neutron security-group-rule-create --protocol icmpv6 --direction ingress --ethertype IPv6 SG-11
neutron security-group-rule-create --protocol icmpv6 --direction egress --ethertype IPv6 SG-11

neutron security-group-rule-create --direction egress --protocol udp --port_range_min 30000 --port_range_max 30500 --ethertype IPv6 SG-111
neutron security-group-rule-create --direction ingress --protocol udp --port_range_min 20000 --port_range_max 20500 --ethertype IPv6 SG-111

neutron security-group-rule-create --direction egress --protocol tcp --port_range_min 37000 --port_range_max 37500 --ethertype IPv6 SG-111
neutron security-group-rule-create --direction ingress --protocol tcp --port_range_min 37000 --port_range_max 37500 --ethertype IPv6 SG-111

neutron security-group-rule-create --protocol icmpv6 --direction ingress --ethertype IPv6 SG-111
neutron security-group-rule-create --protocol icmpv6 --direction egress --ethertype IPv6 SG-111

nova boot --flavor m1.small --image cirros-0.3.4-x86_64-uec --availability-zone nova:test-ovs --nic net-name=N1 SG-21 --security-group=SG-1
nova boot --flavor m1.small --image cirros-0.3.4-x86_64-uec --availability-zone nova:test-ovs --nic net-name=N1 SG-22 --security-group=SG-11
nova boot --flavor m1.small --image cirros-0.3.4-x86_64-uec --availability-zone nova:test-ovs --nic net-name=N1 SG-23 --security-group=SG-11

nova boot --flavor m1.small --image cirros-0.3.4-x86_64-uec --availability-zone nova:test-ovs --nic net-name=N1 SG-24 --security-group=SG-111
nova boot --flavor m1.small --image cirros-0.3.4-x86_64-uec --availability-zone nova:test-ovs --nic net-name=N1 SG-25 --security-group=SG-111

nova boot --flavor m1.small --image cirros-0.3.4-x86_64-uec --availability-zone nova:test-ovs --nic net-name=N1 SG-26 --security-group=SG-1
nova boot --flavor m1.small --image cirros-0.3.4-x86_64-uec --availability-zone nova:test-ovs --nic net-name=N1 SG-27 --security-group=SG-11

stack@test-ovs:~/devstack$ neutron security-group-rule-list
----------------------------------------------------------------------------------------------------+

----------------------------------------------------------------------------------------------------+

----------------------------------------------------------------------------------------------------+

stack@test-ovs:~/devstack$ neutron net-list
-------------------------------------------------------------------------------------------------

-------------------------------------------------------------------------------------------------

-------------------------------------------------------------------------------------------------
stack@test-ovs:~/devstack$ neutron router-list
--------------------------------------------------------------------------+

--------------------------------------------------------------------------+

--------------------------------------------------------------------------+
stack@test-ovs:~/devstack$ neutron subnet-list
-------------------------------------------------------------------------------------------------------------------+

-------------------------------------------------------------------------------------------------------------------+

-------------------------------------------------------------------------------------------------------------------+
stack@test-ovs:~/devstack$

stack@test-ovs:~/devstack$ nova list
------------------------------------------------------------------------------------------------------------+

------------------------------------------------------------------------------------------------------------+

------------------------------------------------------------------------------------------------------------+
stack@test-ovs:~/devstack$ neutron port-ist
Unknown command [u'port-ist']
stack@test-ovs:~/devstack$ neutron port-list
------------------------------------------------------------------------------------------------------------------+

------------------------------------------------------------------------------------------------------------------+

------------------------------------------------------------------------------------------------------------------+
stack@test-ovs:~/devstack$

> ^C
root@test-ovs:~# ovs-vsctl show
1ec09262-5efe-4425-86d7-c6fc1c84b89c
Manager "tcp:10.164.224.227:6640"
is_connected: true
Bridge br-ex
Port br-ex
Interface br-ex
type: internal
Bridge br-int
Controller "tcp:10.164.224.227:6653"
is_connected: true
fail_mode: secure
Port br-int
Interface br-int
type: internal
Port "tap115195f1-56"
Interface "tap115195f1-56"
Port "tap0deb4a0c-71"
Interface "tap0deb4a0c-71"
Port "tapda53afde-da"
Interface "tapda53afde-da"
Port "tap63a2aea6-0f"
Interface "tap63a2aea6-0f"
Port "tapcd22f2f4-d2"
Interface "tapcd22f2f4-d2"
Port "tapd935c2fd-d9"
Interface "tapd935c2fd-d9"
Port "tap1de8ab8a-ef"
Interface "tap1de8ab8a-ef"
Port "tap482e14d1-b9"
Interface "tap482e14d1-b9"
type: internal
ovs_version: "2.6.1"
root@test-ovs:~#

stack@test-ovs:~/devstack$ neutron net-list
-------------------------------------------------------------------------------------------------

-------------------------------------------------------------------------------------------------

-------------------------------------------------------------------------------------------------
stack@test-ovs:~/devstack$ neutron router-list
--------------------------------------------------------------------------+

--------------------------------------------------------------------------+

--------------------------------------------------------------------------+
stack@test-ovs:~/devstack$ neutron subnet-list
-------------------------------------------------------------------------------------------------------------------+

-------------------------------------------------------------------------------------------------------------------+

-------------------------------------------------------------------------------------------------------------------+
stack@test-ovs:~/devstack$
stack@test-ovs:~/devstack$
stack@test-ovs:~/devstack$ nova list
------------------------------------------------------------------------------------------------------------+

------------------------------------------------------------------------------------------------------------+

------------------------------------------------------------------------------------------------------------+
stack@test-ovs:~/devstack$ nova delete SG-21 SG-22 SG-23 SG-24
Request to delete server SG-21 has been accepted.
Request to delete server SG-22 has been accepted.
Request to delete server SG-23 has been accepted.
Request to delete server SG-24 has been accepted.
stack@test-ovs:~/devstack$ nova delete SG-25 SG-26 SG-27
Request to delete server SG-25 has been accepted.
Request to delete server SG-26 has been accepted.
Request to delete server SG-27 has been accepted.
stack@test-ovs:~/devstack$
stack@test-ovs:~/devstack$
stack@test-ovs:~/devstack$ nova list
-------------------------------------------+

-------------------------------------------+
-------------------------------------------+
stack@test-ovs:~/devstack$ neutron port-list
--------------------------------------------------------------------------------------------------------------------+

--------------------------------------------------------------------------------------------------------------------+

on:neutron/ports
Response Body
{
"ports": {
"port": [
{
"uuid": "dcc568ab-4c50-46e9-872b-35870e4c9e11",
"device-id": "1f090594-d8ca-4c97-8c6f-01e3a3f61dba",
"tenant-id": "ded9c85d-092e-41e7-8985-465b627cb28c",
"network-id": "3ea3ed6b-a42a-4ef5-a75f-0e420bd11853",
"neutron-portsecurity:port-security-enabled": true,
"device-owner": "network:router_interface",
"fixed-ips": [

{ "subnet-id": "a7e122e8-d516-4846-806e-19e8bed3302c", "ip-address": "2001:db8:cafe::1" }

],
"neutron-binding:host-id": "",
"neutron-binding:vif-type": "unbound",
"neutron-binding:vnic-type": "normal",
"status": "ACTIVE",
"mac-address": "fa:16:3e:04:7f:8d",
"name": "",
"admin-state-up": true
},
{
"uuid": "482e14d1-b954-4ce9-a468-3e3add9ccb03",
"device-id": "dhcp20a46f74-5aa3-5ff5-bd77-c0be7b92622b-3ea3ed6b-a42a-4ef5-a75f-0e420bd11853",
"tenant-id": "ded9c85d-092e-41e7-8985-465b627cb28c",
"network-id": "3ea3ed6b-a42a-4ef5-a75f-0e420bd11853",
"neutron-portsecurity:port-security-enabled": false,
"device-owner": "network:dhcp",
"fixed-ips": [

{ "subnet-id": "a7e122e8-d516-4846-806e-19e8bed3302c", "ip-address": "2001:db8:cafe:0:f816:3eff:fed4:b6f1" }

],
"neutron-binding:host-id": "test-ovs",
"neutron-binding:vif-details": [

{ "details-key": "port_filter", "value": "true" }

],
"neutron-binding:vif-type": "ovs",
"neutron-binding:vnic-type": "normal",
"mac-address": "fa:16:3e:d4:b6:f1",
"name": "",
"admin-state-up": true
}
]
}
}

Flow table:

root@test-ovs:~# sudo ovs-ofctl dump-flows br-int -OOpenflow13
OFPST_FLOW reply (OF1.3) (xid=0x2):
cookie=0x8000000, duration=1173.309s, table=0, n_packets=9, n_bytes=726, priority=4,in_port=2 actions=write_metadata:0x10000000000/0xffffff0000000001,goto_table:17
cookie=0x8040000, duration=1171.729s, table=17, n_packets=6, n_bytes=468, priority=7,metadata=0x7000010000000000/0xffffff0000000000 actions=write_metadata:0x8000011389000000/0xfffffffffffffffe,goto_table:48
cookie=0x4000000, duration=1133.527s, table=17, n_packets=9, n_bytes=726, priority=4,metadata=0x10000000000/0xffffff0000000000 actions=write_metadata:0x6000011389000000/0xfffffffffffffffe,goto_table:45
cookie=0x8000001, duration=1133.526s, table=17, n_packets=0, n_bytes=0, priority=6,metadata=0x6000010000000000/0xffffff0000000000 actions=write_metadata:0x70000100000222e0/0xfffffffffffffffe,goto_table:19
cookie=0x4000000, duration=969.097s, table=17, n_packets=6, n_bytes=1219, priority=4,metadata=0x4000040000000000/0xffffff0000000000 actions=write_metadata:0x6000041389000000/0xfffffffffffffffe,goto_table:45
cookie=0x4000000, duration=945.293s, table=17, n_packets=2, n_bytes=662, priority=4,metadata=0x4000050000000000/0xffffff0000000000 actions=write_metadata:0x6000051389000000/0xfffffffffffffffe,goto_table:45
cookie=0x8000000, duration=1787.416s, table=17, n_packets=0, n_bytes=0, priority=0,metadata=0x6000000000000000/0xf000000000000000 actions=write_metadata:0x7000000000000000/0xf000000000000000,goto_table:80
cookie=0x6800000, duration=1787.456s, table=18, n_packets=0, n_bytes=0, priority=0 actions=goto_table:38
cookie=0x1080000, duration=1787.097s, table=19, n_packets=0, n_bytes=0, priority=100,arp,arp_op=1 actions=group:5000
cookie=0x1080000, duration=1787.097s, table=19, n_packets=0, n_bytes=0, priority=100,arp,arp_op=2 actions=CONTROLLER:65535,resubmit(,17)
cookie=0x8000009, duration=1133.629s, table=19, n_packets=0, n_bytes=0, priority=20,metadata=0x222e0/0xfffffffe,dl_dst=fa:16:3e:04:7f:8d actions=goto_table:21
cookie=0x1080000, duration=1787.096s, table=19, n_packets=27, n_bytes=6431, priority=0 actions=resubmit(,17)
cookie=0x8000002, duration=1132.397s, table=20, n_packets=0, n_bytes=0, priority=10,mpls,mpls_label=70002 actions=pop_mpls:0x0800,group:150000
cookie=0x1030000, duration=1787.096s, table=20, n_packets=0, n_bytes=0, priority=0 actions=goto_table:80
cookie=0x8000003, duration=1132.388s, table=21, n_packets=0, n_bytes=0, priority=138,ipv6,metadata=0x222e0/0xfffffffe,ipv6_dst=2001:db8:cafe:0:f816:3eff:fed4:b6f1 actions=group:150000
cookie=0x8000004, duration=1787.115s, table=22, n_packets=0, n_bytes=0, priority=0 actions=CONTROLLER:65535
cookie=0x9001389, duration=1171.883s, table=36, n_packets=0, n_bytes=0, priority=5,tun_id=0x1389 actions=write_actions(group:210001)
cookie=0x9000001, duration=1171.882s, table=36, n_packets=0, n_bytes=0, priority=5,tun_id=0x1 actions=load:0x100->NXM_NX_REG6[],resubmit(,220)
cookie=0x9011172, duration=1132.397s, table=36, n_packets=0, n_bytes=0, priority=5,tun_id=0x11172 actions=group:150000
cookie=0x9051389, duration=1173.198s, table=38, n_packets=0, n_bytes=0, priority=5,tun_id=0x403 actions=write_metadata:0x1389000000/0xfffffffff000000,goto_table:51
cookie=0x6900000, duration=961.275s, table=40, n_packets=2, n_bytes=662, priority=63010,udp,metadata=0x50000000000/0xfffff0000000000,tp_src=68,tp_dst=67 actions=resubmit(,17)
cookie=0x6900000, duration=961.172s, table=40, n_packets=0, n_bytes=0, priority=63010,udp6,metadata=0x50000000000/0xfffff0000000000,tp_src=546,tp_dst=547 actions=resubmit(,17)
cookie=0x6900000, duration=961.172s, table=40, n_packets=0, n_bytes=0, priority=63010,udp,metadata=0x50000000000/0xfffff0000000000,tp_src=67,tp_dst=68 actions=drop
cookie=0x6900000, duration=961.123s, table=40, n_packets=0, n_bytes=0, priority=63010,udp6,metadata=0x50000000000/0xfffff0000000000,tp_src=547,tp_dst=546 actions=drop
cookie=0x6900000, duration=960.862s, table=40, n_packets=0, n_bytes=0, priority=63020,icmp6,metadata=0x50000000000/0xfffff0000000000,icmp_type=134,icmp_code=0 actions=drop
cookie=0x6900000, duration=960.814s, table=40, n_packets=0, n_bytes=0, priority=63010,icmp6,metadata=0x50000000000/0xfffff0000000000,icmp_type=133,icmp_code=0 actions=resubmit(,17)
cookie=0x6900000, duration=960.603s, table=40, n_packets=0, n_bytes=0, priority=63010,icmp6,metadata=0x50000000000/0xfffff0000000000,icmp_type=135,icmp_code=0 actions=resubmit(,17)
cookie=0x6900000, duration=960.479s, table=40, n_packets=0, n_bytes=0, priority=63010,icmp6,metadata=0x50000000000/0xfffff0000000000,icmp_type=136,icmp_code=0 actions=resubmit(,17)
cookie=0x6900000, duration=960.252s, table=40, n_packets=0, n_bytes=0, priority=63010,arp,metadata=0x50000000000/0xfffff0000000000,arp_sha=fa:16:3e:62:89:82 actions=resubmit(,17)
cookie=0x6900000, duration=960.067s, table=40, n_packets=0, n_bytes=0, priority=61010,ipv6,dl_src=fa:16:3e:62:89:82,ipv6_src=2001:db8:cafe:0:f816:3eff:fe62:8982 actions=ct(table=41,zone=5001)
cookie=0x6900000, duration=960.021s, table=40, n_packets=0, n_bytes=0, priority=61010,ipv6,dl_src=fa:16:3e:62:89:82,ipv6_src=fe80::f816:3eff:fe62:8982 actions=ct(table=41,zone=5001)
cookie=0x6900000, duration=1787.481s, table=40, n_packets=4, n_bytes=360, priority=0 actions=drop
cookie=0x6900000, duration=1787.478s, table=41, n_packets=0, n_bytes=0, priority=62020,ct_state=-new+est-rel-inv+trk actions=resubmit(,17)
cookie=0x6900000, duration=1787.467s, table=41, n_packets=0, n_bytes=0, priority=62020,ct_state=-new-est+rel-inv+trk actions=resubmit(,17)
cookie=0x6900000, duration=1787.467s, table=41, n_packets=0, n_bytes=0, priority=62020,ct_state=+inv+trk actions=drop
cookie=0x6900000, duration=959.837s, table=41, n_packets=0, n_bytes=0, priority=1005,ct_state=+new+trk,udp6,metadata=0x50000000000/0xfffff0000000000,tp_dst=0x7720/0xfffc actions=ct(commit,zone=5001),resubmit(,17)
cookie=0x6900000, duration=958.223s, table=41, n_packets=0, n_bytes=0, priority=1005,ct_state=+new+trk,tcp6,metadata=0x50000000000/0xfffff0000000000,tp_dst=0x9278/0xfffc actions=ct(commit,zone=5001),resubmit(,17)
cookie=0x6900000, duration=959.528s, table=41, n_packets=0, n_bytes=0, priority=1005,ct_state=+new+trk,udp6,metadata=0x50000000000/0xfffff0000000000,tp_dst=0x7540/0xffc0 actions=ct(commit,zone=5001),resubmit(,17)
cookie=0x6900000, duration=958.441s, table=41, n_packets=0, n_bytes=0, priority=1005,ct_state=+new+trk,tcp6,metadata=0x50000000000/0xfffff0000000000,tp_dst=0x90c0/0xffc0 actions=ct(commit,zone=5001),resubmit(,17)
cookie=0x6900000, duration=957.966s, table=41, n_packets=0, n_bytes=0, priority=1005,ct_state=+new+trk,tcp6,metadata=0x50000000000/0xfffff0000000000,tp_dst=0x9200/0xffc0 actions=ct(commit,zone=5001),resubmit(,17)
cookie=0x6900000, duration=959.336s, table=41, n_packets=0, n_bytes=0, priority=1005,ct_state=+new+trk,udp6,metadata=0x50000000000/0xfffff0000000000,tp_dst=0x7600/0xff00 actions=ct(commit,zone=5001),resubmit(,17)
cookie=0x6900000, duration=957.530s, table=41, n_packets=0, n_bytes=0, priority=1005,ct_state=+new+trk,tcp6,metadata=0x50000000000/0xfffff0000000000,tp_dst=0x9100/0xff00 actions=ct(commit,zone=5001),resubmit(,17)
cookie=0x6900000, duration=959.160s, table=41, n_packets=0, n_bytes=0, priority=1005,ct_state=+new+trk,udp6,metadata=0x50000000000/0xfffff0000000000,tp_dst=0x7700/0xffe0 actions=ct(commit,zone=5001),resubmit(,17)
cookie=0x6900000, duration=958.801s, table=41, n_packets=0, n_bytes=0, priority=1005,ct_state=+new+trk,tcp6,metadata=0x50000000000/0xfffff0000000000,tp_dst=0x90a0/0xffe0 actions=ct(commit,zone=5001),resubmit(,17)
cookie=0x6900000, duration=957.526s, table=41, n_packets=0, n_bytes=0, priority=1005,ct_state=+new+trk,tcp6,metadata=0x50000000000/0xfffff0000000000,tp_dst=0x9240/0xffe0 actions=ct(commit,zone=5001),resubmit(,17)
cookie=0x6900000, duration=959.156s, table=41, n_packets=0, n_bytes=0, priority=1005,ct_state=+new+trk,udp6,metadata=0x50000000000/0xfffff0000000000,tp_dst=0x7580/0xff80 actions=ct(commit,zone=5001),resubmit(,17)
cookie=0x6900000, duration=958.985s, table=41, n_packets=0, n_bytes=0, priority=1005,ct_state=+new+trk,udp6,metadata=0x50000000000/0xfffff0000000000,tp_dst=30500 actions=ct(commit,zone=5001),resubmit(,17)
cookie=0x6900000, duration=957.394s, table=41, n_packets=0, n_bytes=0, priority=1005,ct_state=+new+trk,tcp6,metadata=0x50000000000/0xfffff0000000000,tp_dst=37500 actions=ct(commit,zone=5001),resubmit(,17)
cookie=0x6900000, duration=958.900s, table=41, n_packets=0, n_bytes=0, priority=1005,ct_state=+new+trk,udp6,metadata=0x50000000000/0xfffff0000000000,tp_dst=0x7530/0xfff0 actions=ct(commit,zone=5001),resubmit(,17)
cookie=0x6900000, duration=958.091s, table=41, n_packets=0, n_bytes=0, priority=1005,ct_state=+new+trk,tcp6,metadata=0x50000000000/0xfffff0000000000,tp_dst=0x9260/0xfff0 actions=ct(commit,zone=5001),resubmit(,17)
cookie=0x6900000, duration=957.840s, table=41, n_packets=0, n_bytes=0, priority=1005,ct_state=+new+trk,tcp6,metadata=0x50000000000/0xfffff0000000000,tp_dst=0x9090/0xfff0 actions=ct(commit,zone=5001),resubmit(,17)
cookie=0x6900000, duration=958.710s, table=41, n_packets=0, n_bytes=0, priority=1005,ct_state=+new+trk,tcp6,metadata=0x50000000000/0xfffff0000000000,tp_dst=0x9270/0xfff8 actions=ct(commit,zone=5001),resubmit(,17)
cookie=0x6900000, duration=958.333s, table=41, n_packets=0, n_bytes=0, priority=1005,ct_state=+new+trk,tcp6,metadata=0x50000000000/0xfffff0000000000,tp_dst=0x9088/0xfff8 actions=ct(commit,zone=5001),resubmit(,17)
cookie=0x6900000, duration=957.238s, table=41, n_packets=0, n_bytes=0, priority=1005,ct_state=+new+trk,icmp6,metadata=0x50000000000/0xfffff0000000000 actions=ct(commit,zone=5001),resubmit(,17)
cookie=0x6900000, duration=1787.467s, table=41, n_packets=0, n_bytes=0, priority=50,ct_state=+new+trk actions=drop
cookie=0x6900000, duration=1787.480s, table=41, n_packets=0, n_byte...