The adjacency and learned entries for external subnet GW are not removed in DS after deleting the external subnet. As a result flows for external GW and PNF are not installed after re-creating the external subnet and attaching it to router. Here the ArpNotificationHandler ignores the ARP responses for these external IP addresses since the ip addresses are already learned and stored in DS (with wrong external subnet id)

Release tested: master/oxygen

Reproduction steps: 

1) Create network net1/subnet1
2) Create router router1
3) Attach subnet1 to router
3) Create external network/subnet
  openstack network create ext-net1 --external --provider-physical-network physnet1 --provider-network-type flat
  openstack subnet create --network ext-net1 --allocation-pool start=192.168.56.9,end=192.168.56.20 --no-dhcp --gateway 192.168.56.1 --subnet-range 192.168.56.0/24 ext-subnet1
4) Set external subnet to router
openstack router set --external-gateway ext-net1 router1

Fib flows:
  cookie=0x8000003, duration=10.015s, table=21, n_packets=0, n_bytes=0, priority=42,ip,metadata=0x30d4a/0xfffffe,nw_dst=192.168.56.18 actions=write_metadata:0x30d4a/0xfffffe,goto_table:44
  cookie=0x8000003, duration=5.017s, table=21, n_packets=0, n_bytes=0, priority=42,ip,metadata=0x30d4a/0xfffffe,nw_dst=192.168.56.1 actions=set_field:52:54:00:34:10:b5->eth_dst,load:0x300->NXM_NX_REG6[],resubmit(,220)
  cookie=0x8000003, duration=10.008s, table=21, n_packets=0, n_bytes=0, priority=34,ip,metadata=0x30d4a/0xfffffe,nw_dst=192.168.56.0/24 actions=write_metadata:0x138b030d4a/0xfffffffffe,goto_table:22
  cookie=0x8000004, duration=16.832s, table=21, n_packets=0, n_bytes=0, priority=10,ip,metadata=0x30d4a/0xfffffe actions=group:225000
  cookie=0x8000004, duration=10.005s, table=21, n_packets=0, n_bytes=0, priority=10,ip,metadata=0x30d40/0xfffffe actions=goto_table:26
  cookie=0x8000004, duration=10.008s, table=22, n_packets=0, n_bytes=0, priority=42,ip,metadata=0x30d4a/0xfffffe,nw_dst=192.168.56.255 actions=drop

Adjacencies:

{
  "name": "262580518925378:br-physnet1-pa:trunk",
  "odl-l3vpn:adjacency": [
   

{       "adjacency_type": "primary-adjacency",       "ip_address": "192.168.56.1/32",       "mac_address": "52:54:00:34:10:b5",       "phys-network-func": true,       "subnet_id": "cb843374-e6a4-44a1-a711-f4b6735a5d49"     }

  ],
  "router-interface": false,
  "scheduled-for-remove": false,
  "vpn-instance-names": [
 

{     "associated-subnet-type": "V4AndV6Subnets",     "vpn-name": "2cb27917-216b-4edf-b90a-572e3d53b654"    }

 ]

}

Learned VPN-VIP-TO-PORT

"learnt-vpn-vip-to-port": [

{   "creation-time": "05/31/2018 1:18:52 PM",   "mac-address": "52:54:00:34:10:b5",   "port-fixedip": "192.168.56.1",   "port-name": "262580518925378:br-physnet1-pa:trunk",   "vpn-name": "2cb27917-216b-4edf-b90a-572e3d53b654" }

]

5) Unset external subnet to router
6) Delete external subnet

• FIB flows for external sunet are gone
• However in DS the adjacencies and learned vpn-vip for the external subnet GW remained (subnet_id: cb843374-e6a4-44a1-a711-f4b6735a5d49)

7) Re-create the external subnet
  openstack subnet create --network ext-net1 --allocation-pool start=192.168.56.9,end=192.168.56.20 --no-dhcp --gateway 192.168.56.1 --subnet-range 192.168.56.0/24 ext-subnet1

 

8) Re-Set external subnet to router
openstack router set --external-gateway ext-net1 router1

Issues:

• FIB flows for external subnet GW (192.168.56.1) is not installed
    cookie=0x8000003, duration=10.015s, table=21, n_packets=0, n_bytes=0, priority=42,ip,metadata=0x30d4a/0xfffffe,nw_dst=192.168.56.18 actions=write_metadata:0x30d4a/0xfffffe,goto_table:44
    cookie=0x8000003, duration=10.008s, table=21, n_packets=0, n_bytes=0, priority=34,ip,metadata=0x30d4a/0xfffffe,nw_dst=192.168.56.0/24 actions=write_metadata:0x138b030d4a/0xfffffffffe,goto_table:22
    cookie=0x8000004, duration=16.832s, table=21, n_packets=0, n_bytes=0, priority=10,ip,metadata=0x30d4a/0xfffffe actions=group:225000
    cookie=0x8000004, duration=10.005s, table=21, n_packets=0, n_bytes=0, priority=10,ip,metadata=0x30d40/0xfffffe actions=goto_table:26
    cookie=0x8000004, duration=10.008s, table=22, n_packets=0, n_bytes=0, priority=42,ip,metadata=0x30d4a/0xfffffe,nw_dst=192.168.56.255 actions=drop

• The Adjacency and the learned entries for the deleted external subnet (in step 6) still remains. These entries cause ArpNotificationHandler to ignore
  ArpResponse when re-creating the external subnet. As a result, the adjacency for the re-created external subnet is not added in the DS.