Operating System: All
Platform: All

Attachment karaf_and_flows.zip has been added with description: Logs attached

Issue is most of the time reproducible if i create l3vpn through REST and associate network with it.Logs already provided.

Topology Used: (I am not very good in putting box. So, Will explain the topology used like below)

N-> Network
S-> Subnet
P-> Port(VM)

N1->S1->P1( IP: 10.1.1.2 , MAC: fa:16:3e:d8:0f:0d)
N1->S1->p2( IP: 10.1.1.3 , MAC: fa:16:3e:13:86:a0)

N2->S2->P3( IP: 10.1.2.2 , MAC: fa:16:3e:38:53:66)
N2->S2->P4( IP: 10.1.2.3 , MAC: fa:16:3e:a5:ed:30)

N3->S3->P5( IP: 10.1.3.2 , MAC: fa:16:3e:b1:74:16)
N3->S3->P6( IP: 10.1.3.3 , MAC: fa:16:3e:94:c7:f8)

N4->S4->P7( IP: 10.1.4.2 , MAC: fa:16:3e:99:82:3f)
N4->S4->P8( IP: 10.1.4.3 , MAC: fa:16:3e:9d:b4:1f )

Number of DPN used :2
Tunnel Type Used : GRE between DPN’s

Port1, Port3, Port5 and Port7 booted in DPN1
Port2, Port4, Port6 and Port8 booted in DPN2

Number of VPN Created: 2(abcdabcd-1111-1111-1111-abcdabcdabcd and abcdabcd-2222-2222-2222-abcdabcdabcd)

Command Used in CLI:

Subnet S1 and S2 attached with VPN1

Subnet S3 and S4 attached with VPN2

vpnservice:configure-l3vpn -op create-l3-vpn --subnet-uuid 5fb09a85-6788-4487-a124-078810b8fbcb --vpnid abcdabcd-1111-1111-1111-abcdabcdabcd --name testVpn1 --rd 200:1 --import-rts 200:1 --export-rts 200:1
vpnservice:configure-l3vpn -op create-l3-vpn --subnet-uuid 8d701dc7-332a-43f7-b851-b02fd80fff84 --vpnid abcdabcd-1111-1111-1111-abcdabcdabcd --name testVpn1 --rd 200:1 --import-rts 200:1 --export-rts 200:1
vpnservice:configure-l3vpn -op create-l3-vpn --subnet-uuid 28656a8d-605a-4b9d-b571-a803ebd0f4d9 --vpnid abcdabcd-2222-2222-2222-abcdabcdabcd --name testVpn1 --rd 100:1 --import-rts 100:1 --export-rts 100:1
vpnservice:configure-l3vpn -op create-l3-vpn --subnet-uuid 3567916c-1058-4a6f-8dcb-f9db42d3fab2 --vpnid abcdabcd-2222-2222-2222-abcdabcdabcd --name testVpn1 --rd 100:1 --import-rts 100:1 --export-rts 100:1

Observations:

After creation of VPN ping is successful across N1 to N2 and N3 to N4.
Ex: 10.1.1.2 to 10.1.1.3 , 10.1.2.2 and 10.1.2.3
10.1.3.2 to 10.1.3.3, 10.1.4.2 and 10.1.4.3

Please find the Dumpflows and Groups Outpu Before Deleting VPN.

Before Deleting VPN
Table 21 Entries in DP1:
cookie=0x1030000, duration=3566.559s, table=20, n_packets=0, n_bytes=0, priority=0 actions=goto_table:80
cookie=0x8000003, duration=417.586s, table=21, n_packets=4, n_bytes=392, priority=42,ip,metadata=0x64/0xffffffff,nw_dst=10.1.1.3 actions=write_actions(set_field:0x65->tun_id,output:1)
cookie=0x8000003, duration=413.567s, table=21, n_packets=3, n_bytes=294, priority=42,ip,metadata=0x64/0xffffffff,nw_dst=10.1.1.2 actions=write_actions(group:150001)
cookie=0x8000003, duration=404.480s, table=21, n_packets=3, n_bytes=294, priority=42,ip,metadata=0x64/0xffffffff,nw_dst=10.1.2.2 actions=write_actions(group:150002)
cookie=0x8000003, duration=400.418s, table=21, n_packets=4, n_bytes=392, priority=42,ip,metadata=0x64/0xffffffff,nw_dst=10.1.2.3 actions=write_actions(set_field:0x68->tun_id,output:1)
cookie=0x8000003, duration=395.800s, table=21, n_packets=2, n_bytes=196, priority=42,ip,metadata=0x69/0xffffffff,nw_dst=10.1.3.3 actions=write_actions(set_field:0x6a->tun_id,output:1)
cookie=0x8000003, duration=387.757s, table=21, n_packets=3, n_bytes=294, priority=42,ip,metadata=0x69/0xffffffff,nw_dst=10.1.3.2 actions=write_actions(group:150005)
cookie=0x8000003, duration=379.587s, table=21, n_packets=3, n_bytes=294, priority=42,ip,metadata=0x69/0xffffffff,nw_dst=10.1.4.3 actions=write_actions(set_field:0x6c->tun_id,output:1)
cookie=0x8000003, duration=375.564s, table=21, n_packets=3, n_bytes=294, priority=42,ip,metadata=0x69/0xffffffff,nw_dst=10.1.4.2 actions=write_actions(group:150007)
DumpGroups in DPN1 :
stack@compute-node1:~/devstack$ sudo ovs-ofctl dump-groups -O Openflow13 br-int
OFPST_GROUP_DESC reply (OF1.3) (xid=0x2):
group_id=150001,type=indirect,bucket=actions=set_field:fa:16:3e:d8:0f:0d->eth_dst,output:8
group_id=209999,type=all,bucket=actions=set_field:0x8->tun_id,resubmit(,55)
group_id=150007,type=indirect,bucket=actions=set_field:fa:16:3e:99:82:3f->eth_dst,output:11
group_id=210003,type=all,bucket=actions=set_field:0x4->tun_id,resubmit(,55)
group_id=150005,type=indirect,bucket=actions=set_field:fa:16:3e:b1:74:16->eth_dst,output:10
group_id=210000,type=all,bucket=actions=set_field:0x8->tun_id,resubmit(,55),bucket=actions=set_field:0x1388->tun_id,output:1
group_id=210006,type=all,bucket=actions=set_field:0x6->tun_id,resubmit(,55),bucket=actions=set_field:0x138b->tun_id,output:1
group_id=210004,type=all,bucket=actions=set_field:0x4->tun_id,resubmit(,55),bucket=actions=set_field:0x138a->tun_id,output:1
group_id=150002,type=indirect,bucket=actions=set_field:fa:16:3e:38:53:66->eth_dst,output:9
group_id=210002,type=all,bucket=actions=set_field:0xa->tun_id,resubmit(,55),bucket=actions=set_field:0x1389->tun_id,output:1
group_id=210001,type=all,bucket=actions=set_field:0xa->tun_id,resubmit(,55)
group_id=210005,type=all,bucket=actions=set_field:0x6->tun_id,resubmit(,55)

Flows in DPN2:
cookie=0x8000003, duration=417.586s, table=21, n_packets=4, n_bytes=392, priority=42,ip,metadata=0x64/0xffffffff,nw_dst=10.1.1.3 actions=write_actions(set_field:0x65->tun_id,output:1)
cookie=0x8000003, duration=413.567s, table=21, n_packets=3, n_bytes=294, priority=42,ip,metadata=0x64/0xffffffff,nw_dst=10.1.1.2 actions=write_actions(group:150000)
cookie=0x8000003, duration=404.480s, table=21, n_packets=3, n_bytes=294, priority=42,ip,metadata=0x64/0xffffffff,nw_dst=10.1.2.2 actions=write_actions(group:150003)
cookie=0x8000003, duration=400.418s, table=21, n_packets=4, n_bytes=392, priority=42,ip,metadata=0x64/0xffffffff,nw_dst=10.1.2.3 actions=write_actions(set_field:0x68->tun_id,output:1)
cookie=0x8000003, duration=692.447s, table=21, n_packets=0, n_bytes=0, priority=42,ip,metadata=0x69/0xffffffff,nw_dst=10.1.3.3 actions=write_actions(group:150004)
cookie=0x8000003, duration=688.370s, table=21, n_packets=5, n_bytes=490, priority=42,ip,metadata=0x69/0xffffffff,nw_dst=10.1.3.2 actions=write_actions(set_field:0x6b->tun_id,output:1)
cookie=0x8000003, duration=680.269s, table=21, n_packets=0, n_bytes=0, priority=42,ip,metadata=0x69/0xffffffff,nw_dst=10.1.4.3 actions=write_actions(group:150006)
cookie=0x8000003, duration=676.207s, table=21, n_packets=0, n_bytes=0, priority=42,ip,metadata=0x69/0xffffffff,nw_dst=10.1.4.2 actions=write_actions(set_field:0x6d->tun_id,output:1)
Dumpgroups:
stack@compute-node2:~/devstack$ sudo ovs-ofctl dump-groups -O Openflow13 br-int
OFPST_GROUP_DESC reply (OF1.3) (xid=0x2):
group_id=150004,type=indirect,bucket=actions=set_field:fa:16:3e:94:c7:f8->eth_dst,output:10
group_id=209999,type=all,bucket=actions=set_field:0x9->tun_id,resubmit(,55)
group_id=150006,type=indirect,bucket=actions=set_field:fa:16:3e:9d:b4:1f->eth_dst,output:11
group_id=210003,type=all,bucket=actions=set_field:0x5->tun_id,resubmit(,55)
group_id=210000,type=all,bucket=actions=set_field:0x9->tun_id,resubmit(,55),bucket=actions=set_field:0x1388->tun_id,output:1
group_id=150003,type=indirect,bucket=actions=set_field:fa:16:3e:a5:ed:30->eth_dst,output:9
group_id=210006,type=all,bucket=actions=set_field:0x7->tun_id,resubmit(,55),bucket=actions=set_field:0x138b->tun_id,output:1
group_id=210004,type=all,bucket=actions=set_field:0x5->tun_id,resubmit(,55),bucket=actions=set_field:0x138a->tun_id,output:1
group_id=150000,type=indirect,bucket=actions=set_field:fa:16:3e:13:86:a0->eth_dst,output:8
group_id=210001,type=all,bucket=actions=set_field:0xb->tun_id,resubmit(,55)
group_id=210002,type=all,bucket=actions=set_field:0xb->tun_id,resubmit(,55),bucket=actions=set_field:0x1389->tun_id,output:1
group_id=210005,type=all,bucket=actions=set_field:0x7->tun_id,resubmit(,55)

Now, Delete VPN and Recreate the VPN.

DPN1 Output:

stack@compute-node1:~/devstack$ sudo ovs-ofctl dump-flows -O Openflow13 br-int | grep -i "table=21"
cookie=0x8000003, duration=9318.894s, table=21, n_packets=2568, n_bytes=251664, priority=42,ip,metadata=0x65/0xffffffff,nw_dst=10.1.1.3 actions=write_actions(set_field:0x64->tun_id,output:1)
cookie=0x8000003, duration=9310.852s, table=21, n_packets=2, n_bytes=196, priority=42,ip,metadata=0x65/0xffffffff,nw_dst=10.1.1.2 actions=write_actions(group:150002)
cookie=0x8000003, duration=9304.323s, table=21, n_packets=53, n_bytes=5194, priority=42,ip,metadata=0x65/0xffffffff,nw_dst=10.1.2.2 actions=write_actions(group:150001)
cookie=0x8000003, duration=9300.265s, table=21, n_packets=124, n_bytes=12152, priority=42,ip,metadata=0x65/0xffffffff,nw_dst=10.1.2.3 actions=write_actions(set_field:0x6b->tun_id,output:1)
cookie=0x8000003, duration=9294.919s, table=21, n_packets=256, n_bytes=25088, priority=42,ip,metadata=0x66/0xffffffff,nw_dst=10.1.3.3 actions=write_actions(set_field:0x6d->tun_id,output:1)
cookie=0x8000003, duration=9290.897s, table=21, n_packets=2, n_bytes=196, priority=42,ip,metadata=0x66/0xffffffff,nw_dst=10.1.3.2 actions=write_actions(group:150005)
cookie=0x8000003, duration=9284.985s, table=21, n_packets=435, n_bytes=42630, priority=42,ip,metadata=0x66/0xffffffff,nw_dst=10.1.4.3 actions=write_actions(set_field:0x69->tun_id,output:1)
cookie=0x8000003, duration=9280.967s, table=21, n_packets=2, n_bytes=196, priority=42,ip,metadata=0x66/0xffffffff,nw_dst=10.1.4.2 actions=write_actions(group:150006)
cookie=0x8000003, duration=16124.832s, table=21, n_packets=2500, n_bytes=233744, priority=0 actions=goto_table:80
stack@compute-node1:~/devstack$ sudo ovs-ofctl dump-groups -O Openflow13 br-int
OFPST_GROUP_DESC reply (OF1.3) (xid=0x2):
group_id=150001,type=indirect,bucket=actions=set_field:fa:16:3e:d8:0f:0d->eth_dst,output:8  Mismatch in MAC address. It should be : fa:16:3e:38:53:66
group_id=209999,type=all,bucket=actions=set_field:0x8->tun_id,resubmit(,55)
group_id=150006,type=indirect,bucket=actions=set_field:fa:16:3e:99:82:3f->eth_dst,output:11
group_id=150007,type=indirect,bucket=actions=set_field:fa:16:3e:99:82:3f->eth_dst,output:11
group_id=210003,type=all,bucket=actions=set_field:0x4->tun_id,resubmit(,55)
group_id=150005,type=indirect,bucket=actions=set_field:fa:16:3e:b1:74:16->eth_dst,output:10
group_id=210000,type=all,bucket=actions=set_field:0x8->tun_id,resubmit(,55),bucket=actions=set_field:0x1388->tun_id,output:1
group_id=150003,type=indirect,bucket=actions=set_field:fa:16:3e:d8:0f:0d->eth_dst,output:8
group_id=210006,type=all,bucket=actions=set_field:0x6->tun_id,resubmit(,55),bucket=actions=set_field:0x138b->tun_id,output:1
group_id=210004,type=all,bucket=actions=set_field:0x4->tun_id,resubmit(,55),bucket=actions=set_field:0x138a->tun_id,output:1
group_id=150000,type=indirect,bucket=actions=set_field:fa:16:3e:38:53:66->eth_dst,output:9
group_id=150002,type=indirect,bucket=actions=set_field:fa:16:3e:38:53:66->eth_dst,output:9  Mismatch in MAC address. It should be fa:16:3e:d8:0f:0d
group_id=210002,type=all,bucket=actions=set_field:0xa->tun_id,resubmit(,55),bucket=actions=set_field:0x1389->tun_id,output:1
group_id=210001,type=all,bucket=actions=set_field:0xa->tun_id,resubmit(,55)
group_id=210005,type=all,bucket=actions=set_field:0x6->tun_id,resubmit(,55)

DPN2 Output:
stack@compute-node2:~/devstack$ sudo ovs-ofctl dump-flows -O Openflow13 br-int | grep -i "table=21"
cookie=0x8000003, duration=9383.279s, table=21, n_packets=0, n_bytes=0, priority=42,ip,metadata=0x65/0xffffffff,nw_dst=10.1.1.3 actions=write_actions(group:150003)
cookie=0x8000003, duration=9379.237s, table=21, n_packets=0, n_bytes=0, priority=42,ip,metadata=0x65/0xffffffff,nw_dst=10.1.1.2 actions=write_actions(set_field:0x68->tun_id,output:1)
cookie=0x8000003, duration=9372.702s, table=21, n_packets=0, n_bytes=0, priority=42,ip,metadata=0x65/0xffffffff,nw_dst=10.1.2.2 actions=write_actions(set_field:0x67->tun_id,output:1)
cookie=0x8000003, duration=9368.677s, table=21, n_packets=0, n_bytes=0, priority=42,ip,metadata=0x65/0xffffffff,nw_dst=10.1.2.3 actions=write_actions(group:150000)
cookie=0x8000003, duration=9363.334s, table=21, n_packets=0, n_bytes=0, priority=42,ip,metadata=0x66/0xffffffff,nw_dst=10.1.3.3 actions=write_actions(group:150004)
cookie=0x8000003, duration=9359.273s, table=21, n_packets=347, n_bytes=34006, priority=42,ip,metadata=0x66/0xffffffff,nw_dst=10.1.3.2 actions=write_actions(set_field:0x6c->tun_id,output:1)
cookie=0x8000003, duration=9353.398s, table=21, n_packets=0, n_bytes=0, priority=42,ip,metadata=0x66/0xffffffff,nw_dst=10.1.4.3 actions=write_actions(group:150007)
cookie=0x8000003, duration=9349.348s, table=21, n_packets=344, n_bytes=33712, priority=42,ip,metadata=0x66/0xffffffff,nw_dst=10.1.4.2 actions=write_actions(set_field:0x6a->tun_id,output:1)
cookie=0x8000003, duration=16131.578s, table=21, n_packets=45, n_bytes=1890, priority=0 actions=goto_table:80
stack@compute-node2:~/devstack$ sudo ovs-ofctl dump-groups -O Openflow13 br-int
OFPST_GROUP_DESC reply (OF1.3) (xid=0x2):
group_id=150001,type=indirect,bucket=actions=set_field:fa:16:3e:13:86:a0->eth_dst,output:8
group_id=150004,type=indirect,bucket=actions=set_field:fa:16:3e:94:c7:f8->eth_dst,output:10
group_id=209999,type=all,bucket=actions=set_field:0x9->tun_id,resubmit(,55)
group_id=150007,type=indirect,bucket=actions=set_field:fa:16:3e:9d:b4:1f->eth_dst,output:11
group_id=150006,type=indirect,bucket=actions=set_field:fa:16:3e:9d:b4:1f->eth_dst,output:11  Duplicate entry
group_id=210003,type=all,bucket=actions=set_field:0x5->tun_id,resubmit(,55)
group_id=210000,type=all,bucket=actions=set_field:0x9->tun_id,resubmit(,55),bucket=actions=set_field:0x1388->tun_id,output:1
group_id=150003,type=indirect,bucket=actions=set_field:fa:16:3e:a5:ed:30->eth_dst,output:9  Mismatch in MAC Address,MAC Address should be: :fa:16:3e:13:86:a0
group_id=210006,type=all,bucket=actions=set_field:0x7->tun_id,resubmit(,55),bucket=actions=set_field:0x138b->tun_id,output:1
group_id=210004,type=all,bucket=actions=set_field:0x5->tun_id,resubmit(,55),bucket=actions=set_field:0x138a->tun_id,output:1
group_id=150002,type=indirect,bucket=actions=set_field:fa:16:3e:a5:ed:30->eth_dst,output:9  Duplicate entry.
group_id=150000,type=indirect,bucket=actions=set_field:fa:16:3e:13:86:a0->eth_dst,output:8  Mismatch in MAC Address. MAC Address should be: :fa:16:3e:a5:ed:30
group_id=210001,type=all,bucket=actions=set_field:0xb->tun_id,resubmit(,55)
group_id=210002,type=all,bucket=actions=set_field:0xb->tun_id,resubmit(,55),bucket=actions=set_field:0x1389->tun_id,output:1
group_id=210005,type=all,bucket=actions=set_field:0x7->tun_id,resubmit(,55)

So, ping failed with below combination.
10.1.1.2 to 10.1.1.3, 10.1.2.2 and 10.1.2.3 is failed .
10.1.2.2 to 10.1.1.2,10.1.1.3 and 10.1.2.4 is failed.
But,Below ping is successful.
10.1.3.2 to 10.1.3.3,10.1.4.2 and 10.1.4.3 is successful
10.1.4.2 to 10.1.3.2,10.1.3.3 and 10.1.4.4 is successful

This is based on my observation and understanding. Please correct me if my analysis is wrong and let me know how I can proceed further.

So, Basically packet is reaching another VM properly. Due, to mismatch in MAC address it is getting dropped there and no reply is going back again.

Please find the more Karaf logs attached with this mail.

Note: Above is tested with creation and deletion of VPN through CLI(SF137 Scope). Here reproducing this issue is not always happing.

I have tested the same behavior by creating L3VPN through REST and Associate network with it using Neutron.(This will be tested in SF141. But, since neutron code is already available I am validating this test case here too.
REST USED to Create VPN:
http://10.183.255.21:8181/restconf/operations/neutronvpn:createL3VPN
{"input" : {
"l3vpn" : [

,

]
}
}
http://10.183.255.21:8181/restconf/operations/neutronvpn:associateNetworks
{"input" :

}

Assume same topology is used. But, with NEW MAC address.

10.1.1.2 fa:16:3e:2f:c3:88
10.1.1.3 fa:16:3e:fa:4c:fc
10.1.2.2 fa:16:3e:5b:8d:86
10.1.2.3 fa:16:3e:f4:cd:fd
10.1.3.2 fa:16:3e:9b:b5:07
10.1.3.3 fa:16:3e:ba:2f:3e
10.1.4.2 fa:16:3e:22:d6:e0
10.1.4.3 fa:16:3e:54:ef:ca

Please find the logs before deletion of VPN.
DPN1:
stack@compute-node1:~/devstack$ cat before_deletion
cookie=0x8000003, duration=210.777s, table=21, n_packets=0, n_bytes=0, priority=42,ip,metadata=0x64/0xffffffff,nw_dst=10.1.1.3 actions=write_actions(set_field:0x66->tun_id,output:1)
cookie=0x8000003, duration=202.755s, table=21, n_packets=0, n_bytes=0, priority=42,ip,metadata=0x64/0xffffffff,nw_dst=10.1.1.2 actions=write_actions(group:150001)
cookie=0x8000003, duration=191.613s, table=21, n_packets=0, n_bytes=0, priority=42,ip,metadata=0x64/0xffffffff,nw_dst=10.1.2.2 actions=write_actions(group:150002)
cookie=0x8000003, duration=187.560s, table=21, n_packets=0, n_bytes=0, priority=42,ip,metadata=0x64/0xffffffff,nw_dst=10.1.2.3 actions=write_actions(set_field:0x69->tun_id,output:1)
cookie=0x8000003, duration=154.927s, table=21, n_packets=0, n_bytes=0, priority=42,ip,metadata=0x65/0xffffffff,nw_dst=10.1.3.2 actions=write_actions(group:150004)
cookie=0x8000003, duration=150.882s, table=21, n_packets=0, n_bytes=0, priority=42,ip,metadata=0x65/0xffffffff,nw_dst=10.1.3.3 actions=write_actions(set_field:0x6b->tun_id,output:1)
cookie=0x8000003, duration=141.983s, table=21, n_packets=0, n_bytes=0, priority=42,ip,metadata=0x65/0xffffffff,nw_dst=10.1.4.3 actions=write_actions(set_field:0x6c->tun_id,output:1)
cookie=0x8000003, duration=137.958s, table=21, n_packets=0, n_bytes=0, priority=42,ip,metadata=0x65/0xffffffff,nw_dst=10.1.4.2 actions=write_actions(group:150007)
cookie=0x8000003, duration=814.172s, table=21, n_packets=0, n_bytes=0, priority=0 actions=goto_table:80
OFPST_GROUP_DESC reply (OF1.3) (xid=0x2):
group_id=150004,type=indirect,bucket=actions=set_field:fa:16:3e:9b:b5:07->eth_dst,output:4
group_id=150001,type=indirect,bucket=actions=set_field:fa:16:3e:2f:c3:88->eth_dst,output:2
group_id=209999,type=all,bucket=actions=set_field:0x4->tun_id,resubmit(,55)
group_id=150007,type=indirect,bucket=actions=set_field:fa:16:3e:22:d6:e0->eth_dst,output:5
group_id=210003,type=all,bucket=actions=set_field:0x8->tun_id,resubmit(,55)
group_id=210000,type=all,bucket=actions=set_field:0x4->tun_id,resubmit(,55),bucket=actions=set_field:0x1388->tun_id,output:1
group_id=210006,type=all,bucket=actions=set_field:0xa->tun_id,resubmit(,55),bucket=actions=set_field:0x138b->tun_id,output:1
group_id=210004,type=all,bucket=actions=set_field:0x8->tun_id,resubmit(,55),bucket=actions=set_field:0x138a->tun_id,output:1
group_id=150002,type=indirect,bucket=actions=set_field:fa:16:3e:5b:8d:86->eth_dst,output:3
group_id=210001,type=all,bucket=actions=set_field:0x6->tun_id,resubmit(,55)
group_id=210002,type=all,bucket=actions=set_field:0x6->tun_id,resubmit(,55),bucket=actions=set_field:0x1389->tun_id,output:1
group_id=210005,type=all,bucket=actions=set_field:0xa->tun_id,resubmit(,55)

DPN2:
stack@compute-node2:~/devstack$ cat before_deletion
cookie=0x8000003, duration=611.523s, table=21, n_packets=0, n_bytes=0, priority=42,ip,metadata=0x64/0xffffffff,nw_dst=10.1.1.3 actions=write_actions(group:150000)
cookie=0x8000003, duration=607.443s, table=21, n_packets=5, n_bytes=490, priority=42,ip,metadata=0x64/0xffffffff,nw_dst=10.1.1.2 actions=write_actions(set_field:0x67->tun_id,output:1)
cookie=0x8000003, duration=596.296s, table=21, n_packets=0, n_bytes=0, priority=42,ip,metadata=0x64/0xffffffff,nw_dst=10.1.2.2 actions=write_actions(set_field:0x68->tun_id,output:1)
cookie=0x8000003, duration=592.275s, table=21, n_packets=0, n_bytes=0, priority=42,ip,metadata=0x64/0xffffffff,nw_dst=10.1.2.3 actions=write_actions(group:150003)
cookie=0x8000003, duration=563.641s, table=21, n_packets=0, n_bytes=0, priority=42,ip,metadata=0x65/0xffffffff,nw_dst=10.1.3.2 actions=write_actions(set_field:0x6a->tun_id,output:1)
cookie=0x8000003, duration=555.599s, table=21, n_packets=0, n_bytes=0, priority=42,ip,metadata=0x65/0xffffffff,nw_dst=10.1.3.3 actions=write_actions(group:150005)
cookie=0x8000003, duration=546.701s, table=21, n_packets=0, n_bytes=0, priority=42,ip,metadata=0x65/0xffffffff,nw_dst=10.1.4.3 actions=write_actions(group:150006)
cookie=0x8000003, duration=542.639s, table=21, n_packets=0, n_bytes=0, priority=42,ip,metadata=0x65/0xffffffff,nw_dst=10.1.4.2 actions=write_actions(set_field:0x6d->tun_id,output:1)
cookie=0x8000003, duration=1170.456s, table=21, n_packets=2, n_bytes=84, priority=0 actions=goto_table:80
OFPST_GROUP_DESC reply (OF1.3) (xid=0x2):
group_id=209999,type=all,bucket=actions=set_field:0x5->tun_id,resubmit(,55)
group_id=150006,type=indirect,bucket=actions=set_field:fa:16:3e:54:ef:ca->eth_dst,output:5
group_id=210003,type=all,bucket=actions=set_field:0x9->tun_id,resubmit(,55)
group_id=150005,type=indirect,bucket=actions=set_field:fa:16:3e:ba:2f:3e->eth_dst,output:4
group_id=210000,type=all,bucket=actions=set_field:0x5->tun_id,resubmit(,55),bucket=actions=set_field:0x1388->tun_id,output:1
group_id=150003,type=indirect,bucket=actions=set_field:fa:16:3e:f4:cd:fd->eth_dst,output:3
group_id=210004,type=all,bucket=actions=set_field:0x9->tun_id,resubmit(,55),bucket=actions=set_field:0x138a->tun_id,output:1
group_id=210006,type=all,bucket=actions=set_field:0xb->tun_id,resubmit(,55),bucket=actions=set_field:0x138b->tun_id,output:1
group_id=150000,type=indirect,bucket=actions=set_field:fa:16:3e:fa:4c:fc->eth_dst,output:2
group_id=210002,type=all,bucket=actions=set_field:0x7->tun_id,resubmit(,55),bucket=actions=set_field:0x1389->tun_id,output:1
group_id=210001,type=all,bucket=actions=set_field:0x7->tun_id,resubmit(,55)
group_id=210005,type=all,bucket=actions=set_field:0xb->tun_id,resubmit(,55)

After Deletion and recreation of VPN(+ Associating Network with VPN)

DPN1:
stack@compute-node1:~/devstack$ cat after_deletion
cookie=0x8000003, duration=152.415s, table=21, n_packets=0, n_bytes=0, priority=42,ip,metadata=0x67/0xffffffff,nw_dst=10.1.1.3 actions=write_actions(set_field:0x66->tun_id,output:1)
cookie=0x8000003, duration=144.367s, table=21, n_packets=0, n_bytes=0, priority=42,ip,metadata=0x67/0xffffffff,nw_dst=10.1.1.2 actions=write_actions(group:150002)
cookie=0x8000003, duration=129.001s, table=21, n_packets=0, n_bytes=0, priority=42,ip,metadata=0x67/0xffffffff,nw_dst=10.1.2.2 actions=write_actions(group:150000)
cookie=0x8000003, duration=124.932s, table=21, n_packets=7, n_bytes=686, priority=42,ip,metadata=0x67/0xffffffff,nw_dst=10.1.2.3 actions=write_actions(set_field:0x6a->tun_id,output:1)
cookie=0x8000003, duration=112.590s, table=21, n_packets=0, n_bytes=0, priority=42,ip,metadata=0x68/0xffffffff,nw_dst=10.1.3.2 actions=write_actions(group:150004)
cookie=0x8000003, duration=108.542s, table=21, n_packets=0, n_bytes=0, priority=42,ip,metadata=0x68/0xffffffff,nw_dst=10.1.3.3 actions=write_actions(set_field:0x6b->tun_id,output:1)
cookie=0x8000003, duration=98.531s, table=21, n_packets=0, n_bytes=0, priority=42,ip,metadata=0x68/0xffffffff,nw_dst=10.1.4.3 actions=write_actions(set_field:0x6d->tun_id,output:1)
cookie=0x8000003, duration=94.509s, table=21, n_packets=0, n_bytes=0, priority=42,ip,metadata=0x68/0xffffffff,nw_dst=10.1.4.2 actions=write_actions(group:150007)
cookie=0x8000003, duration=1579.002s, table=21, n_packets=4, n_bytes=168, priority=0 actions=goto_table:80
OFPST_GROUP_DESC reply (OF1.3) (xid=0x2):
group_id=150004,type=indirect,bucket=actions=set_field:fa:16:3e:9b:b5:07->eth_dst,output:4
group_id=150001,type=indirect,bucket=actions=set_field:fa:16:3e:2f:c3:88->eth_dst,output:2
group_id=209999,type=all,bucket=actions=set_field:0x4->tun_id,resubmit(,55)
group_id=150007,type=indirect,bucket=actions=set_field:fa:16:3e:22:d6:e0->eth_dst,output:5
group_id=210003,type=all,bucket=actions=set_field:0x8->tun_id,resubmit(,55)
group_id=210000,type=all,bucket=actions=set_field:0x4->tun_id,resubmit(,55),bucket=actions=set_field:0x1388->tun_id,output:1
group_id=210006,type=all,bucket=actions=set_field:0xa->tun_id,resubmit(,55),bucket=actions=set_field:0x138b->tun_id,output:1
group_id=210004,type=all,bucket=actions=set_field:0x8->tun_id,resubmit(,55),bucket=actions=set_field:0x138a->tun_id,output:1
group_id=150000,type=indirect,bucket=actions=set_field:fa:16:3e:5b:8d:86->eth_dst,output:3
group_id=150002,type=indirect,bucket=actions=set_field:fa:16:3e:5b:8d:86->eth_dst,output:3  Mismatch in MAC address. MAC address should be : fa:16:3e:2f:c3:88
group_id=210001,type=all,bucket=actions=set_field:0x6->tun_id,resubmit(,55)
group_id=210002,type=all,bucket=actions=set_field:0x6->tun_id,resubmit(,55),bucket=actions=set_field:0x1389->tun_id,output:1
group_id=210005,type=all,bucket=actions=set_field:0xa->tun_id,resubmit(,55)

DPN2:

stack@compute-node2:~/devstack$ cat after_deletion
cookie=0x8000003, duration=116.723s, table=21, n_packets=0, n_bytes=0, priority=42,ip,metadata=0x67/0xffffffff,nw_dst=10.1.1.3 actions=write_actions(group:150001)
cookie=0x8000003, duration=112.672s, table=21, n_packets=7, n_bytes=686, priority=42,ip,metadata=0x67/0xffffffff,nw_dst=10.1.1.2 actions=write_actions(set_field:0x69->tun_id,output:1)
cookie=0x8000003, duration=97.309s, table=21, n_packets=0, n_bytes=0, priority=42,ip,metadata=0x67/0xffffffff,nw_dst=10.1.2.2 actions=write_actions(set_field:0x64->tun_id,output:1)
cookie=0x8000003, duration=93.274s, table=21, n_packets=0, n_bytes=0, priority=42,ip,metadata=0x67/0xffffffff,nw_dst=10.1.2.3 actions=write_actions(group:150003)
cookie=0x8000003, duration=84.926s, table=21, n_packets=0, n_bytes=0, priority=42,ip,metadata=0x68/0xffffffff,nw_dst=10.1.3.2 actions=write_actions(set_field:0x6c->tun_id,output:1)
cookie=0x8000003, duration=76.879s, table=21, n_packets=0, n_bytes=0, priority=42,ip,metadata=0x68/0xffffffff,nw_dst=10.1.3.3 actions=write_actions(group:150006)
cookie=0x8000003, duration=66.871s, table=21, n_packets=0, n_bytes=0, priority=42,ip,metadata=0x68/0xffffffff,nw_dst=10.1.4.3 actions=write_actions(group:150005)
cookie=0x8000003, duration=62.815s, table=21, n_packets=0, n_bytes=0, priority=42,ip,metadata=0x68/0xffffffff,nw_dst=10.1.4.2 actions=write_actions(set_field:0x65->tun_id,output:1)
cookie=0x8000003, duration=1498.909s, table=21, n_packets=3, n_bytes=126, priority=0 actions=goto_table:80
OFPST_GROUP_DESC reply (OF1.3) (xid=0x2):
group_id=150001,type=indirect,bucket=actions=set_field:fa:16:3e:fa:4c:fc->eth_dst,output:2
group_id=209999,type=all,bucket=actions=set_field:0x5->tun_id,resubmit(,55)
group_id=150006,type=indirect,bucket=actions=set_field:fa:16:3e:54:ef:ca->eth_dst,output:5  Mismatch in MAC address. MAC address should be : fa:16:3e:ba:2f:3e
group_id=210003,type=all,bucket=actions=set_field:0x9->tun_id,resubmit(,55)
group_id=150005,type=indirect,bucket=actions=set_field:fa:16:3e:ba:2f:3e->eth_dst,output:4  Mismatch in MAC address. MAC address should be: fa:16:3e:54:ef:ca.
group_id=210000,type=all,bucket=actions=set_field:0x5->tun_id,resubmit(,55),bucket=actions=set_field:0x1388->tun_id,output:1
group_id=150003,type=indirect,bucket=actions=set_field:fa:16:3e:f4:cd:fd->eth_dst,output:3
group_id=210004,type=all,bucket=actions=set_field:0x9->tun_id,resubmit(,55),bucket=actions=set_field:0x138a->tun_id,output:1
group_id=210006,type=all,bucket=actions=set_field:0xb->tun_id,resubmit(,55),bucket=actions=set_field:0x138b->tun_id,output:1
group_id=150000,type=indirect,bucket=actions=set_field:fa:16:3e:fa:4c:fc->eth_dst,output:2
group_id=210002,type=all,bucket=actions=set_field:0x7->tun_id,resubmit(,55),bucket=actions=set_field:0x1389->tun_id,output:1
group_id=210001,type=all,bucket=actions=set_field:0x7->tun_id,resubmit(,55)
group_id=210005,type=all,bucket=actions=set_field:0xb->tun_id,resubmit(,55)

With the Second test, Observation is:

On VPN1 : - Ping from 10.1.1.2 to remining VM’s failed and from other VM’s we can not ping 10.1.1.2 Due to mismatch in MAC address.
On VPN2: - ping from 10.1.3.3 and 10.1.4.2 to other VM’s failed due to mismatch in MAC address.

Attachment ping_isse_after_vpn_deletion.zip has been added with description: karaf logs attached

vpnservice is no longer active