Failure in VTN Flowfilter CSIT to install flow with actions vtn-set-inet-src and vtn-set-inet-dst

VTN Flowfilter CSIT which failed to install flows with actions vtn-set-inet-src and vtn-set-inet-dst in Boron.
Failure is not consistent. And issue occurs in OF13.

Followed below steps in creating VTN Flowfilter test case
1) Create VTN
2) Create VBR1
3) Create VBR1-IF1 and VBR1-IF2
4) Configure portmap s2-eth1 (host h1) for VBR1-IF1
5) Configure portmap s3-eth1 (host h3) for VBR1-IF2
6) ping h1 and h3
7) Create VBR2
8) Create VBR2-IF3 and VBR2-IF4
9) Configure portmap s2-eth2 (host h2) for VBR2-IF3
10 Configure portmap s3-eth2 (host h4) for VBR2-IF4
11 ping h2 and h4
12 Create flowcondition with vtn-flow-math [inet4-src 10.0.0.1/32 and inet4-dst 10.0.0.3/32]
13) Create VTN Flowfilter with vtn-set-inet-src (192.0.0.1/32) and vtn-set-inet-dst (192.0.0.3/32)
14) Verify Mininet ping should not succeed Between h1 and h3.
15) Verify flowentry installed with vtn-set-inet-src (192.0.0.1) and vtn-set-inet-dst (192.0.0.3) -> Here test case fails to detect inet-src and inet-dst in flow entry installed.

https://logs.opendaylight.org/releng/jenkins092/vtn-csit-1node-manager-only-boron/649/archives/karaf.log.gz

Below are the observation from the above log and Steps of VTN flowfilter

1) When the above Step-13 is executed, found path-fault issue between openflow:2 --> openflow:3. And vbridge1 status become DOWN.
Please find log below for the same.

2016-09-05 16:30:59,731 | TRACE | Runner: VTN Main | VTNFlowCondition | 194 - org.opendaylight.vtn.manager.implementation - 0.5.0.SNAPSHOT | cond_1: Matched the condition: match=[DL_TYPE=2048,IP_SRC=10.0.0.1,IP_DST=10.0.0.3], packet=[Ether[src=56:33:cb:a7:29:f5,dst=6a:ef:a7:a4:36:f2,type=0x800],Inet4[src=10.0.0.1,dst=10.0.0.3,proto=1,dscp=0],ICMP[type=8,code=0]]
2016-09-05 16:30:59,731 | TRACE | Runner: VTN Main | VTNPassFilter | 194 - org.opendaylight.vtn.manager.implementation - 0.5.0.SNAPSHOT | VTN:Tenant1%IN.1: Flow action was applied: VTNSetInetSrcAction[addr=192.0.0.1,order=1]
2016-09-05 16:30:59,731 | TRACE | Runner: VTN Main | VTNPassFilter | 194 - org.opendaylight.vtn.manager.implementation - 0.5.0.SNAPSHOT | VTN:Tenant1%IN.1: Flow action was applied: VTNSetInetDstAction[addr=192.0.0.2,order=2]
2016-09-05 16:30:59,731 | TRACE | Runner: VTN Main | VTNPassFilter | 194 - org.opendaylight.vtn.manager.implementation - 0.5.0.SNAPSHOT | VTN:Tenant1%IN.1: Packet matched the condition: cond_1
2016-09-05 16:30:59,733 | WARN | n-dispatcher-149 | VTenantManager | 194 - org.opendaylight.vtn.manager.implementation - 0.5.0.SNAPSHOT | vBridge:Tenant1/vBridge1: Path fault: openflow:2 -> openflow:3
2016-09-05 16:30:59,733 | INFO | n-dispatcher-149 | VTenantManager | 194 - org.opendaylight.vtn.manager.implementation - 0.5.0.SNAPSHOT | vBridge:Tenant1/vBridge1: vBridge status has been changed: old=

{state=UP, path-faults=0}

, new=

{state=DOWN, path-faults=1}

2) Few seconds later (5sec) missing edge added and inter switch link has been created. Which in turns resolves path-fault issue.

2016-09-05 16:31:04,338 | INFO | n-dispatcher-149 | VTNRoutingManager | 194 - org.opendaylight.vtn.manager.implementation - 0.5.0.SNAPSHOT | Inter-switch link has been created:

{id=openflow:1:2, src=openflow:1:2, dst=openflow:3:3, static=false}

2016-09-05 16:31:04,338 | TRACE | n-dispatcher-149 | TopologyGraph | 194 - org.opendaylight.vtn.manager.implementation - 0.5.0.SNAPSHOT | Edge added: openflow:1:2 -> openflow:3:3
2016-09-05 16:31:04,338 | INFO | n-dispatcher-149 | VTNRoutingManager | 194 - org.opendaylight.vtn.manager.implementation - 0.5.0.SNAPSHOT | Routing table has been updated.
2016-09-05 16:31:04,338 | INFO | on-dispatcher-45 | VTNInventoryManager | 194 - org.opendaylight.vtn.manager.implementation - 0.5.0.SNAPSHOT | Port has been changed: old={id=openflow:3:3, name=s3-eth3, enabled=true, cost=1000, links=[

{id=openflow:3:3, peer=openflow:1:2}

]}, new={id=openflow:3:3, name=s3-eth3, enabled=true, cost=1000, links=[

{id=openflow:3:3, peer=openflow:1:2}

,

{id=openflow:1:2, peer=openflow:1:2}

]}
2016-09-05 16:31:04,339 | INFO | n-dispatcher-121 | VTNInventoryManager | 194 - org.opendaylight.vtn.manager.implementation - 0.5.0.SNAPSHOT | Port has been changed: old={id=openflow:1:2, name=s1-eth2, enabled=true, cost=1000, links=[

{id=openflow:3:3, peer=openflow:3:3}

]}, new={id=openflow:1:2, name=s1-eth2, enabled=true, cost=1000, links=[

{id=openflow:3:3, peer=openflow:3:3}

,

{id=openflow:1:2, peer=openflow:3:3}

]}

3) But packet doesn't match the condition even after the path-fault resolved.

2016-09-05 16:31:04,675 | TRACE | Runner: VTN Main | VTNFlowCondition | 194 - org.opendaylight.vtn.manager.implementation - 0.5.0.SNAPSHOT | cond_1: Does not match: match=[DL_TYPE=2048,IP_SRC=10.0.0.1,IP_DST=10.0.0.3], packet=[Ether[src=56:33:cb:a7:29:f5,dst=6a:ef:a7:a4:36:f2,type=0x806]]
2016-09-05 16:31:04,675 | TRACE | Runner: VTN Main | VTNFlowCondition | 194 - org.opendaylight.vtn.manager.implementation - 0.5.0.SNAPSHOT | cond_1: Unmatched: packet=[Ether[src=56:33:cb:a7:29:f5,dst=6a:ef:a7:a4:36:f2,type=0x806]]
2016-09-05 16:31:04,675 | TRACE | Runner: VTN Main | VTNPassFilter | 194 - org.opendaylight.vtn.manager.implementation - 0.5.0.SNAPSHOT | VTN:Tenant1%IN.1: Packet did not match the condition: cond_1
2016-09-05 16:31:04,676 | TRACE | Runner: VTN Main | VBridge | 194 - org.opendaylight.vtn.manager.implementation - 0.5.0.SNAPSHOT | 0: Packet route resolved: openflow:2:1 -> [LinkEdge[openflow:2:3 -> openflow:1:1], LinkEdge[openflow:1:2 -> openflow:3:3]] -> openflow:3:1
2016-09-05 16:31:04,676 | TRACE | Runner: VTN Main | VBridge | 194 - org.opendaylight.vtn.manager.implementation - 0.5.0.SNAPSHOT | vBridge:Tenant1/vBridge1: Forward packet to known host: src=56:33:cb:a7:29:f5, dst=6a:ef:a7:a4:36:f2, port=openflow:3:1, type=0x806, vlan=0
2016-09-05 16:31:04,677 | TRACE | Runner: VTN Main | Packet | 192 - org.opendaylight.vtn.manager - 0.5.0.SNAPSHOT | Serialized: ARP: 00:01:08:00:06:04:00:01:56:33:cb:a7:29:f5:0a:00:00:01:00:00:00:00:00:00:0a:00:00:03
2016-09-05 16:31:04,677 | TRACE | Runner: VTN Main | Packet | 192 - org.opendaylight.vtn.manager - 0.5.0.SNAPSHOT | Serialized: Ethernet: 6a:ef:a7:a4:36:f2:56:33:cb:a7:29:f5:08:06:00:01:08:00:06:04:00:01:56:33:cb:a7:29:f5:0a:00:00:01:00:00:00:00:00:00:0a:00:00:03
2016-09-05 16:31:04,677 | TRACE | entLoopGroup-5-5 | VTNPacketService | 194 - org.opendaylight.vtn.manager.implementation - 0.5.0.SNAPSHOT | transmit-packet: RPC completed successfully: result=null

4) Due to unmatched packet, flow entry installed without vtn-set-inet-src and vtn-set-inet-dst in openflow3:1

2016-09-05 16:31:04,684 | TRACE | entLoopGroup-5-5 | SendBarrierRpc | 194 - org.opendaylight.vtn.manager.implementation - 0.5.0.SNAPSHOT | send-barrier RPC has completed: node=openflow:3
2016-09-05 16:31:04,684 | TRACE | TN Flow Thread-0 | FlowAddContext | 194 - org.opendaylight.vtn.manager.implementation - 0.5.0.SNAPSHOT | Flow entry has been installed: flow=[id=7f56000000000027-2, pri=11, timeout=(0,0), node=openflow:3, ingress=openflow:3:3, cond=

{DL_SRC=56:33:cb:a7:29:f5,DL_DST=6a:ef:a7:a4:36:f2,DL_TYPE=2054,DL_VLAN=0}

, actions=

{OUTPUT(port=openflow:3:1, len=65535)}

We will add Report_Failure_Due_To_Bug in the failed test case.
Which report that a test failed due to a known Bugzilla bug whose Bug id is provided as an argument.