NAPT_PFIB_TABLE (table 47) flows for external network/subnet are incorrectly removed
when one of routers unset from the external subnet which still has some other routers
associating with it.

Version Tested: flourine/master

NAT mode: controller

The scenario is illustrated as below:

                          ----------------
                          | External Net |
                          ----------------
                                  |
                                  |
                         -------------------
                         | External Subnet |
                         -------------------
                             |          |
                        ------          ------
                        |                    |
                   -----------          -----------
                   | Router1 |          | Router2 |
                   -----------          -----------
                        |                     |
                        |                     |
                   ----------           ----------
                   | Subnet1 |           | Subnet2 |
                   ----------           ----------

Reproduction Steps:

1) Create External Network/Subnet1

openstack network create ext-net1 --external --provider-physical-network physnet1 --provider-network-type flat

openstack subnet create --network ext-net1 --allocation-pool start=192.168.56.9,end=192.168.56.20 --no-dhcp --gateway 192.168.56.1 --subnet-range 192.168.56.0/24 ext-subnet1

2) Create Router1, set External Subnet1 as gateway

openstack router create router1

3) Create Router2, set External Subnet1 as gateway

openstack router create router2

4) Creat Net/Subnet1

openstack network create vx-net1 --provider-network-type vxlan --provider-segment 1501
openstack subnet create --network vx-net1 --subnet-range 10.100.1.0/24 --dns-nameserver 8.8.8.8 --allocation-pool start=10.100.1.5,end=10.100.1.100 vx-subnet1

5) Create VM on Subnet1

openstack server create --flavor m1.nano --image cirros-0.3.5-x86_64-disk --nic net-id=vx-net1 vm11 --availability-zone=nova:control --key-name admin_key

6) Attach Subnet1 to Router1

openstack router add subnet router1 vx-subnet1

The following NAPT_PFIB flows are installed in table 47. The first flow is for router1's vpn, the 2nd/3rd flows are for external subnet abd external network's
vpn respectively:

cookie=0x8000006, duration=151.068s, table=47, n_packets=5, n_bytes=541, priority=5,ip,metadata=0x30d40/0xfffffe actions=load:0->NXM_OF_IN_PORT[],resubmit(,21) [1]
cookie=0x8000006, duration=49.138s, table=47, n_packets=8, n_bytes=680, priority=5,ip,metadata=0x30d4c/0xfffffe actions=load:0->NXM_OF_IN_PORT[],resubmit(,21) [2]
cookie=0x8000006, duration=49.138s, table=47, n_packets=0, n_bytes=0, priority=5,ip,metadata=0x30d4a/0xfffffe actions=load:0->NXM_OF_IN_PORT[],resubmit(,21) [3]

7) Create Net/Subnet2

openstack network create vx-net2 --provider-network-type vxlan --provider-segment 1502
openstack subnet create --network vx-net2 --subnet-range 10.100.2.0/24 --dns-nameserver 8.8.8.8 --allocation-pool start=10.100.2.5,end=10.100.2.100 vx-subnet2

8) Create VM on Subnet2

openstack server create --flavor m1.nano --image cirros-0.3.5-x86_64-disk --nic net-id=vx-net2 vm21 --availability-zone=nova:control --key-name admin_key\

9) Attach Subnet2 to Router2

openstack router add subnet router2 vx-subnet2

New NAPT_PFIB flow for router2's vpn is installed:

cookie=0x8000006, duration=49.118s, table=47, n_packets=0, n_bytes=0, priority=5,ip,metadata=0x30d54/0xfffffe actions=load:0->NXM_OF_IN_PORT[],resubmit(,21) [4]

10) Verify that VMs from both Subnet1 and Subnet2 can communicate with external network via Router1 and Router2 respectively

ssh from vm11/vm21 to external gateway server (192.168.56.1) 

11) Unset External subnet from router2

openstack router unset --external-gateway router2

---------------------------------------------------
Expected:

• VM from Subnet1 still can communicate with the External Network via Router1
• VM from Subnet2 can not communicate with External Network since Router are unset from Externnal Network

Actual
------

• Failure when Ssh from VMs in BOTH Subnet1 and Subnet2 to servers in External Network

Issue:
NAPT_FIB flows after step 11) Unset Router2 from External Subnet1

cookie=0x8000006, duration=151.068s, table=47, n_packets=5, n_bytes=541, priority=5,ip,metadata=0x30d40/0xfffffe actions=load:0->NXM_OF_IN_PORT[],resubmit(,21)

The NAPT_PFIB_TABLE flows [2] [3] [4] are removed.

• Flow [4] - NAPT_PFIB for Router2's vpn - should be removed is removed.
• However flows [2] [3] - NAPT_PFIB for the External Network1/Subnet1 -
  should not be removed since they are still used by Router1.