All SG Rules getting Removed while removing any one of the SG associated With the VM instance

Created Two Vm instance and attached Two Security groups(sg1,sg2) both having ICMP ingress/egress and TCP ingress/egress.
After removing sg2 from VM instance all the rules getting removed from table 243.

steps to reproduce the issue:
1. Create Security groups,
openstack security group create sg1
openstack security group create sg2

2. Delete default rules from sg1 and sg2

openstack security group rule delete <rule_id_ingress>
openstack security group rule delete <rule_id_egress>

3. Associate rules to SG,

openstack security group rule create --ingress --protocol tcp sg1
openstack security group rule create --ingress --protocol icmp sg1
openstack security group rule create --egress --protocol icmp sg1

openstack security group rule create --ingress --protocol tcp sg2
openstack security group rule create --ingress --protocol icmp --icmp-type 8 --icmp-code 0 sg2
openstack security group rule create --egress --protocol icmp --icmp-type 8 --icmp-code 0 sg2

4. Create Network
openstack network create l2_network_1 --provider-network-type vxlan

openstack subnet create --network l2_network_1 --subnet-range 30.0.0.0/24 l2_subnet_1

5. Create VM
openstack server create --image <imageID> --flavor m1.tiny --nic net-id=l2_network_1 VM1 --security-group sg1

openstack server create --image <imageID> --flavor m1.tiny --nic net-id=l2_network_1 VM2 --security-group sg1
6.Add sg2 to VM
openstack server add security group VM1 sg2
openstack server add security group VM2 sg2

7. Test ping between VM1 and VM2

8. Remove SG2 from VMs

openstack server remove security group VM1 sg2
openstack server remove security group VM2 sg2

9. Test ping between VM1 and VM2

After step 8 unable to login to the VM instance. all the rules getting removed from table 243.

Flows after step 5:
VM1 &VM2 with sg1

cookie=0x6900000, duration=239.553s, table=242, n_packets=4, n_bytes=1352, priority=0 actions=goto_table:243
cookie=0x6900000, duration=239.553s, table=243, n_packets=0, n_bytes=0, priority=62020,ct_state=-new+est-rel-inv+trk actions=resubmit(,220)
cookie=0x6900000, duration=239.553s, table=243, n_packets=0, n_bytes=0, priority=62020,ct_state=-new-est+rel-inv+trk actions=resubmit(,220)
cookie=0x6900001, duration=110.514s, table=243, n_packets=0, n_bytes=0, priority=62015,ct_state=+inv+trk,reg6=0x200/0xfffff00 actions=drop
cookie=0x6900001, duration=88.868s, table=243, n_packets=0, n_bytes=0, priority=62015,ct_state=+inv+trk,reg6=0x500/0xfffff00 actions=drop
cookie=0x6900000, duration=110.514s, table=243, n_packets=0, n_bytes=0, priority=1000,ct_state=+new+trk,icmp,reg6=0x200/0xfffff00 actions=ct(commit,zone=5002),resubmit(,220)
cookie=0x6900000, duration=110.514s, table=243, n_packets=0, n_bytes=0, priority=1001,ct_state=+new+trk,tcp,reg6=0x200/0xfffff00 actions=ct(commit,zone=5002),resubmit(,220)
cookie=0x6900000, duration=88.868s, table=243, n_packets=0, n_bytes=0, priority=1002,ct_state=+new+trk,icmp,reg6=0x500/0xfffff00 actions=ct(commit,zone=5002),resubmit(,220)
cookie=0x6900000, duration=88.868s, table=243, n_packets=0, n_bytes=0, priority=1003,ct_state=+new+trk,tcp,reg6=0x500/0xfffff00 actions=ct(commit,zone=5002),resubmit(,220)
cookie=0x6900001, duration=110.514s, table=243, n_packets=0, n_bytes=0, priority=50,ct_state=+new+trk,reg6=0x200/0xfffff00 actions=drop
cookie=0x6900001, duration=88.868s, table=243, n_packets=0, n_bytes=0, priority=50,ct_state=+new+trk,reg6=0x500/0xfffff00 actions=drop
cookie=0x6900000, duration=239.553s, table=243, n_packets=4, n_bytes=1352, priority=0 actions=drop

Flows after step 6:
VM1 & Vm2 with sg1 &sg2

cookie=0x6900000, duration=770.806s, table=243, n_packets=102, n_bytes=11321, priority=62020,ct_state=-new+est-rel-inv+trk actions=resubmit(,220)
cookie=0x6900000, duration=770.806s, table=243, n_packets=0, n_bytes=0, priority=62020,ct_state=-new-est+rel-inv+trk actions=resubmit(,220)
cookie=0x6900001, duration=641.767s, table=243, n_packets=0, n_bytes=0, priority=62015,ct_state=+inv+trk,reg6=0x200/0xfffff00 actions=drop
cookie=0x6900001, duration=620.121s, table=243, n_packets=0, n_bytes=0, priority=62015,ct_state=+inv+trk,reg6=0x500/0xfffff00 actions=drop
cookie=0x6900000, duration=17.146s, table=243, n_packets=0, n_bytes=0, priority=1004,ct_state=+new+trk,icmp,reg6=0x200/0xfffff00 actions=ct(commit,zone=5002),resubmit(,220)
cookie=0x6900000, duration=17.137s, table=243, n_packets=0, n_bytes=0, priority=1005,ct_state=+new+trk,tcp,reg6=0x200/0xfffff00 actions=ct(commit,zone=5002),resubmit(,220)
cookie=0x6900000, duration=17.129s, table=243, n_packets=0, n_bytes=0, priority=1006,ct_state=+new+trk,tcp,reg6=0x200/0xfffff00 actions=ct(commit,zone=5002),resubmit(,220)
cookie=0x6900000, duration=0.429s, table=243, n_packets=0, n_bytes=0, priority=1007,ct_state=+new+trk,icmp,reg6=0x500/0xfffff00 actions=ct(commit,zone=5002),resubmit(,220)
cookie=0x6900000, duration=0.429s, table=243, n_packets=0, n_bytes=0, priority=1008,ct_state=+new+trk,tcp,reg6=0x500/0xfffff00 actions=ct(commit,zone=5002),resubmit(,220)
cookie=0x6900000, duration=0.429s, table=243, n_packets=0, n_bytes=0, priority=1009,ct_state=+new+trk,tcp,reg6=0x500/0xfffff00 actions=ct(commit,zone=5002),resubmit(,220)
cookie=0x6900001, duration=641.767s, table=243, n_packets=0, n_bytes=0, priority=50,ct_state=+new+trk,reg6=0x200/0xfffff00 actions=drop
cookie=0x6900001, duration=620.121s, table=243, n_packets=0, n_bytes=0, priority=50,ct_state=+new+trk,reg6=0x500/0xfffff00 actions=drop
cookie=0x6900000, duration=770.806s, table=243, n_packets=4, n_bytes=1352, priority=0 actions=drop

Flows after step 6:
removed sg2 from VM1 & VM2

cookie=0x6900000, duration=852.849s, table=243, n_packets=163, n_bytes=18836, priority=62020,ct_state=-new+est-rel-inv+trk actions=resubmit(,220)
cookie=0x6900000, duration=852.849s, table=243, n_packets=0, n_bytes=0, priority=62020,ct_state=-new-est+rel-inv+trk actions=resubmit(,220)
cookie=0x6900001, duration=723.810s, table=243, n_packets=0, n_bytes=0, priority=62015,ct_state=+inv+trk,reg6=0x200/0xfffff00 actions=drop
cookie=0x6900001, duration=702.164s, table=243, n_packets=0, n_bytes=0, priority=62015,ct_state=+inv+trk,reg6=0x500/0xfffff00 actions=drop
cookie=0x6900001, duration=723.810s, table=243, n_packets=0, n_bytes=0, priority=50,ct_state=+new+trk,reg6=0x200/0xfffff00 actions=drop
cookie=0x6900001, duration=702.164s, table=243, n_packets=3, n_bytes=222, priority=50,ct_state=+new+trk,reg6=0x500/0xfffff00 actions=drop
cookie=0x6900000, duration=852.849s, table=243, n_packets=4, n_bytes=1352, priority=0 actions=drop