When remote-sg is applied to the server, flow entries are not removed after server deletion

Setup:

3 control node

2 compute node 

3 controller in cluster 

DIstro: Nitrogen formal release 

Steps to reproduce the issue:

1. Create 2  networks
2. Create VM for each networks
3. Apply ICMP remote SG for the VMs
4. Create a router and add the interfaces for the networks.
5. Check the communication between the VMs
6. Delete the VMs

Observation:

When VMs are deleted, the flows are not removed from the dump-flows. 

Note: Occurs occasionally.