Dynamic SG addition for TCP is taking time to establish connection

We have a three-node neutron-ha cluster.

When adding SG with TCP rule to VM dynamically takes time to establish connection.

Steps to reproduce issue:

openstack network create network_1 --provider-network-type vxlan
openstack subnet create --network network_1 --subnet-range 30.0.0.0/24 l2_subnet_1
Create SG1 without any rules
Create SG2 with ingress and egress tcp rule
openstack server create --image cirros --flavor cirros --nic net-id=network_1 VM1 --security-group SG1
openstack server create --image cirros --flavor cirros --nic net-id=network_1 VM2 --security-group SG1
login to VM1 and start listening (nohup nc -v -l -p 1111 &)
login to VM2 and send data (nc 30.0.0.12 1111) to VM1
On going communication remove SG1 and SG2 to both VM's and send data (takes time to send data)