CSIT for VTN Manager failed to install flows in Boron 3-node job.

https://logs.opendaylight.org/releng/jenkins092/vtn-csit-3node-manager-all-boron/72/archives/log.html.gz

Issue in VTN Dataflow test file. Where we use get-data-flow request to read flows of specified vtn.

Followed below steps in test
1) Create VTN
2) Create VBR1
3) Create VBR1-IF1 and VBR1-IF2
4) Configure portmap s2-eth1 (host h1) for VBR1-IF1
5) Configure portmap s3-eth1 (host h3) for VBR1-IF2
6) ping h1 and h3
7) Verify if data flow exists --> test cases failed in this case
8) Create VBR2
9) Create VBR2-IF3 and VBR2-IF4
10) Configure portmap s2-eth2 (host h2) for VBR2-IF3
11) Configure portmap s3-eth2 (host h4) for VBR2-IF4
12) ping h2 and h4
13) Verify if data flow exists

Logs:
https://logs.opendaylight.org/releng/jenkins092/vtn-csit-3node-manager-all-boron/72/archives/odl1_karaf.log.gz

Observation:

1) After creating portmap for VBR1-(vbridge1), the following error log which has source ip as 10.0.0.3 and destination ip as 10.0.0.1. But in our case sorce ip should be 10.0.0.1 and dst ip 10.0.0.3.
I have verified with success log of same scenario.

016-09-13 13:14:27,311 | TRACE | pool-27-thread-1 | Packet | 192 - org.opendaylight.vtn.manager - 0.5.0.SNAPSHOT | Deserializing: IPv4: SourceIPAddress: 0a:00:00:03 (offset 208 bitsize 32)
2016-09-13 13:14:27,311 | TRACE | pool-27-thread-1 | Packet | 192 - org.opendaylight.vtn.manager - 0.5.0.SNAPSHOT | Deserializing: IPv4: DestinationIPAddress: 0a:00:00:01 (offset 240 bitsize 32)

Before creating portmap s3-eth1 for VBR1-IF2, received remove flow event log as below.

2016-09-13 13:14:27,312 | TRACE | pool-27-thread-1 | VTNFlowManager | 194 - org.opendaylight.vtn.manager.implementation - 0.5.0.SNAPSHOT | Ignore FLOW_REMOVED: Already removed: node=openflow:3, flow=FlowRemoved [_cookie=FlowCookie [_value=9175521290813964347], _match=Match [_ethernetMatch=EthernetMatch [_ethernetDestination=EthernetDestination [_address=MacAddress [_value=da:46:81:12:db:0a], augmentation=[]], _ethernetSource=EthernetSource [_address=MacAddress [_value=aa:a2:cf:0e:d5:0c], augmentation=[]], augmentation=[]], _inPort=Uri [_value=openflow:3:3], _vlanMatch=VlanMatch [_vlanId=VlanId [_vlanId=VlanId [_value=0], _vlanIdPresent=false, augmentation=[]], augmentation=[]], augmentation=[]], _node=NodeRef [_value=KeyedInstanceIdentifier

], _priority=10, _tableId=0, augmentation=[]]
2016-09-13 13:14:27,312 | TRACE | pool-27-thread-1 | VTNFlowManager | 194 - org.opendaylight.vtn.manager.implementation - 0.5.0.SNAPSHOT | Ignore FLOW_REMOVED: Already removed: node=openflow:2, flow=FlowRemoved [_cookie=FlowCookie [_value=9175521290813964347], _match=Match [_ethernetMatch=EthernetMatch [_ethernetDestination=EthernetDestination [_address=MacAddress [_value=da:46:81:12:db:0a], augmentation=[]], _ethernetSource=EthernetSource [_address=MacAddress [_value=aa:a2:cf:0e:d5:0c], augmentation=[]], augmentation=[]], _inPort=Uri [_value=openflow:2:1], _vlanMatch=VlanMatch [_vlanId=VlanId [_vlanId=VlanId [_value=0], _vlanIdPresent=false, augmentation=[]], augmentation=[]], augmentation=[]], _node=NodeRef [_value=KeyedInstanceIdentifier

], _priority=10, _tableId=0, augmentation=[]]
2016-09-13 13:14:27,334 | DEBUG | ner: VTN Flow DS | FlowRemoveContext | 194 - org.opendaylight.vtn.manager.implementation - 0.5.0.SNAPSHOT | Data flow not found: node=openflow:1, flow=59
2016-09-13 13:14:27,334 | DEBUG | ner: VTN Flow DS | FlowRemoveContext | 194 - org.opendaylight.vtn.manager.implementation - 0.5.0.SNAPSHOT | No data flow was removed: remover=flow-removed[id=59, node=openflow:1]

Normally after configuring portmap, there will be ARP packet will be sent from src to dst and dst to src to resolve packet route.
But here in this failure case, ARP request sent from 10.0.0.3 to 0.0.0.0(unknown ip)as following

2016-09-13 13:14:27,350 | TRACE | pool-27-thread-1 | Packet | 192 - org.opendaylight.vtn.manager - 0.5.0.SNAPSHOT | Deserializing: ARP: SenderProtocolAddress: 0a:00:00:03 (offset 224 bitsize 32)
2016-09-13 13:14:27,350 | TRACE | pool-27-thread-1 | Packet | 192 - org.opendaylight.vtn.manager - 0.5.0.SNAPSHOT | Deserializing: ARP: TargetHardwareAddress: bc:76:4e:04:a4:18 (offset 256 bitsize 48)
2016-09-13 13:14:27,350 | TRACE | pool-27-thread-1 | Packet | 192 - org.opendaylight.vtn.manager - 0.5.0.SNAPSHOT | Deserializing: ARP: TargetProtocolAddress: 00:00:00:00 (offset 304 bitsize 32)

Due to the above issues, flowentry is not installed and failed to get flows.