Operating System: All
Platform: All

Thank you for creating this bug report!!!

I've checked your explanation, but it is not clear to me that which behavior of the VTN Manager was wrong. Could you make it clear which behavior of the VTN Manager was wrong? I'm thinking VTN Manager worked fine as expected.

The VTN Manager didn't filter the packet, since the packet didn't match the condition. As the following log message showed, the packet was an ARP packet (type=0x806), but not an IP packet. On the other hand, the conditis for IP packets (DL_TYPE=2048). Therefore, I think the behavior of the VTN Manager was fine.

2016-09-05 16:31:04,675 (snip) cond_1: Does not match: match=[DL_TYPE=2048,IP_SRC=10.0.0.1,IP_DST=10.0.0.3], packet=[Ether[src=56:33:cb:a7:29:f5,dst=6a:ef:a7:a4:36:f2,type=0x806]]
2016-09-05 16:31:04,675 (snip) cond_1: Unmatched: packet=[Ether[src=56:33:cb:a7:29:f5,dst=6a:ef:a7:a4:36:f2,type=0x806]]

The following log message meant that the flow entry was installed for ARP packets (DL_TYPE=2054). The flow entry was not related to the flow filter. Therefore, that's fine.

2016-09-05 16:31:04,684 (snip) Flow entry has been installed: flow=[id=7f56000000000027-2, pri=11, timeout=(0,0), node=openflow:3, ingress=openflow:3:3, cond=

, actions=

Therefore, I could not see any issues of the VTN Manager here. Could you explain which behavior of the VTN Manager was wrong in the test? I'm thinking that the test script of the CSIT may be wrong, although I've not checked the CSIT code deeply.

Or, I'm thinking the path-fault caused the test failure. It could be that the VTN Manager didn't forward the IP packets between h1 and h3, and didn't install flow entries for them due to the path-fault. That could be the reason why the flow entries for the flow filter was not installed. In that case, we need to investigate why the path-fault occurred to check if the path-fault was expected or not.

Attachment FailureCSIT.zip has been added with description: Attached Failure CSIT Karaf log with TRACE enabled for VTN and Openflowplugin

Attachment SuccessCSIT.zip has been added with description: Attached Success CSIT Karaf log with TRACE enabled for VTN and Openflowplugin

To investigate the inconsistent path-fault issue, The following has been added to the CIST test cases.
Collecting VTN-Inventory information
Collecting ODL-Inventory information
Enabling Trace log for Openflow plugin.

A patch with the above changes has been pushed to git.
https://git.opendaylight.org/gerrit/#/c/45617/

The patch was tested in Sandbox and the issue are reproduced.

Attached Karaf log zip file for Failed CSIT build and Success CSIT Build. We compared and analysed both logs to understand the issue.
Attached File Info:
1) Failure CSIT karaf Log - FailureCSIT.zip
2) Success CSIT karaf Log - SuccessCSIT.zip

Analysis:
============
The path-fault is detected during the Verify inet4src and inet4dst of vtn flowfilter test case in the VTN Manager Flowfilter test scenario.
The path-fault is due to the Inter-switch link between ports openflow:1:2 and openflow:3:3 is not present in the VTN Inventory during the test execution.

On analyzing the failure karaf log, The Inter-Switch link between ports openflow:1:2 and openflow:3:3 removal was detected when the configuration was removed in the previous test case ‘Remove Portmap for If1 test case’ in Vtn Manager Dataflow test scenario.

Log Snippet -
Open flow plugin detects Link removal
2016-09-18 19:28:13,049 | DEBUG | yExporter-flow:1 | OperationProcessor | 345 - org.opendaylight.openflowplugin.applications.topology-manager - 0.3.1.SNAPSHOT | New onLinkRemoved operation available, starting transaction

VTN Manager detects link removal
2016-09-18 19:28:13,077 | INFO | on-dispatcher-71 | VTNRoutingManager | 492 - org.opendaylight.vtn.manager.implementation - 0.5.1.SNAPSHOT | Inter-switch link has been removed:

2016-09-18 19:28:13,077 | TRACE | on-dispatcher-71 | TopologyGraph | 492 - org.opendaylight.vtn.manager.implementation - 0.5.1.SNAPSHOT | Edge removed: openflow:1:2 -> openflow:3:3

Also in the get vtn-inventory link edge opneflow:1:2 and openflow:3:3 is missing. And the same link ID is present in other test cases.

VTN-Inventory of Delete VTN Test case in Failure case
=============================================
{"vtn-nodes":{"vtn-node":[{"id":"openflow:3","vtn-port":[{"id":"openflow:3:3","cost":1000,"name":"s3-eth3","enabled":true,"port-link":[

]},

,

],"openflow-version":"OF13"},{"id":"openflow:2","vtn-port":[{"id":"openflow:2:3","cost":1000,"name":"s2-eth3","enabled":true,"port-link":[

,

]},

,

],"openflow-version":"OF13"},{"id":"openflow:1","vtn-port":[{"id":"openflow:1:2","cost":1000,"name":"s1-eth2","enabled":true,"port-link":[

]},{"id":"openflow:1:1","cost":1000,"name":"s1-eth1","enabled":true,"port-link":[

,

]}],"openflow-version":"OF13"}]}}

VTN-Inventory of Delete VTN Test case in Success case.
=============================================
{"vtn-nodes":{"vtn-node":[{"id":"openflow:3","vtn-port":[

,

,{"id":"openflow:3:3","port-link":[

,

],"cost":1000,"name":"s3-eth3","enabled":true}],"openflow-version":"OF13"},{"id":"openflow:2","vtn-port":[{"id":"openflow:2:3","port-link":[

,

],"cost":1000,"name":"s2-eth3","enabled":true},

,

],"openflow-version":"OF13"},{"id":"openflow:1","vtn-port":[{"id":"openflow:1:2","port-link":[

,

],"cost":1000,"name":"s1-eth2","enabled":true},{"id":"openflow:1:1","port-link":[

,

],"cost":1000,"name":"s1-eth1","enabled":true}],"openflow-version":"OF13"}]}}

In the above vtn-inventory,

is missing in node:id ""openflow:3"" and

is missing in node:id ""openflow:1""

On completion of VTN Manager Dataflow Test scenario, VTN Manager Flowfilter test scenario is executed.
Below are Steps followed in VTN Manager Flowfilter test.
1) Create VTN -> pass
2) Create VBR1 -> pass
3) Create VBR1-IF1 and VBR1-IF2 -> pass
4) Configure portmap s2-eth1 (host h1) for VBR1-IF1 -> pass
5) Configure portmap s3-eth1 (host h3) for VBR1-IF2 -> pass
6) ping h1 and h3 -> pass
7) Create VBR2 -> pass
8) Create VBR2-IF3 and VBR2-IF4 -> pass
9) Configure portmap s2-eth2 (host h2) for VBR2-IF3 -> pass
10 Configure portmap s3-eth2 (host h4) for VBR2-IF4 -> pass
11 ping h2 and h4 -> pass
12 Create flowcondition with vtn-flow-math [inet4-src 10.0.0.1/32 and inet4-dst 10.0.0.3/32] -> pass
13) Create VTN Flowfilter with vtn-set-inet-src (192.0.0.1/32) and vtn-set-inet-dst (192.0.0.3/32) -> pass

From step 1 to step 13 the output of VTN-Inventory list doesn't contain Inter-switch link link edge opneflow:1:2 and openflow:3:3.

VTN Inventory Output for the above Step 1 to step 13 - Inter-switch link is missing.
{"vtn-nodes":{"vtn-node":[{"id":"openflow:3","vtn-port":[{"id":"openflow:3:3","cost":1000,"name":"s3-eth3","enabled":true,"port-link":[

]},

,

],"openflow-version":"OF13"},{"id":"openflow:2","vtn-port":[{"id":"openflow:2:3","cost":1000,"name":"s2-eth3","enabled":true,"port-link":[

,

]},

,

],"openflow-version":"OF13"},{"id":"openflow:1","vtn-port":[{"id":"openflow:1:2","cost":1000,"name":"s1-eth2","enabled":true,"port-link":[

]},{"id":"openflow:1:1","cost":1000,"name":"s1-eth1","enabled":true,"port-link":[

,

]}],"openflow-version":"OF13"}]}}

As the Inter-switch link is not present, Step 6 - ping h1 and h3 and Step 11 - ping h2 to h4 should actually fail but the ping is successful in CSIT. In the karaf log VTN Manager is able to successfully resolve the path

2016-09-18 19:28:14,002 | TRACE | Runner: VTN Main | VBridge | 492 - org.opendaylight.vtn.manager.implementation - 0.5.1.SNAPSHOT | 0: Packet route resolved: openflow:3:1 -> [LinkEdge[openflow:3:3 -> openflow:1:2], LinkEdge[openflow:1:1 -> openflow:2:3]] -> openflow:2:1
2016-09-18 19:28:14,466 | TRACE | Runner: VTN Main | VBridge | 492 - org.opendaylight.vtn.manager.implementation - 0.5.1.SNAPSHOT | 0: Packet route resolved: openflow:3:2 -> [LinkEdge[openflow:3:3 -> openflow:1:2], LinkEdge[openflow:1:1 -> openflow:2:3]] -> openflow:2:2

Also we cross verified the same ping test case with the successful CSIT job there in VTN-Inventory list Inter-switch link is present.

14) Verify Mininet ping between h1 and h3. -> pass (ping should not work)
During this ping test VTN Manager detects a path fault.

2016-09-18 19:28:14,707 | WARN | on-dispatcher-71 | VTenantManager | 492 - org.opendaylight.vtn.manager.implementation - 0.5.1.SNAPSHOT | vBridge:Tenant1/vBridge1: Path fault: openflow:2 -> openflow:3
2016-09-18 19:28:14,707 | INFO | on-dispatcher-71 | VTenantManager | 492 - org.opendaylight.vtn.manager.implementation - 0.5.1.SNAPSHOT | vBridge:Tenant1/vBridge1: vBridge status has been changed: old=

, new=

Due to this no flows are added to the switch by the VTN Manager.

15) Verify flowentry installed with vtn-set-inet-src (192.0.0.1) and vtn-set-inet-dst (192.0.0.3) -> As No flows are written this verification test fails in CSIT.

During the execution of test 14 , Openflow plugin detects the link between Openflow:1:2 and Openflow:3:3
2016-09-18 19:28:16,768 | DEBUG | yExporter-flow:1 | OperationProcessor | 345 - org.opendaylight.openflowplugin.applications.topology-manager - 0.3.1.SNAPSHOT | New onLinkDiscovered operation available, starting transaction
2016-09-18 19:28:16,768 | DEBUG | yExporter-flow:1 | OperationProcessor | 345 - org.opendaylight.openflowplugin.applications.topology-manager - 0.3.1.SNAPSHOT | Next operation onLinkDiscovered

VTN Manager Added the link to the Inventory.
2016-09-18 19:28:16,786 | INFO | on-dispatcher-71 | VTNRoutingManager | 492 - org.opendaylight.vtn.manager.implementation - 0.5.1.SNAPSHOT | Inter-switch link has been created:

2016-09-18 19:28:16,786 | TRACE | on-dispatcher-71 | TopologyGraph | 492 - org.opendaylight.vtn.manager.implementation - 0.5.1.SNAPSHOT | Edge added: openflow:1:2 -> openflow:3:3

The missing inter switch links are added to the VTN inventory during step 15 and the subsequent tests are executed successfully in CSIT.

{"vtn-nodes":{"vtn-node":[{"id":"openflow:3","vtn-port":[{"id":"openflow:3:3","cost":1000,"name":"s3-eth3","enabled":true,"port-link":[

,

]},

,

],"openflow-version":"OF13"},{"id":"openflow:2","vtn-port":[{"id":"openflow:2:3","cost":1000,"name":"s2-eth3","enabled":true,"port-link":[

,

]},

,

],"openflow-version":"OF13"},{"id":"openflow:1","vtn-port":[{"id":"openflow:1:2","cost":1000,"name":"s1-eth2","enabled":true,"port-link":[

,

]},{"id":"openflow:1:1","cost":1000,"name":"s1-eth1","enabled":true,"port-link":[

,

]}],"openflow-version":"OF13"}]}}

In the ODL Inventory all the ODL ports are present for all the test cases. We found only the Port information and not link information in the ODL inventory.

Observations -
1) As the Link edges are removed in the earlier test scenario, Ping test in step 6 and 11 should fail in VTN Manager, but the tests are successful and the path is resolved. Why Path fault was not reported during these tests? Is this the expected behavior of VTN Manager?
2) As the Link addition/removal was detected by Openflow plugin first followed by VTN Manager, Our understanding is VTN Manager is handing the Openflow plugin events correctly. Is our understanding correct.
3) In the Openflow plugin logs we didn't find any specific reason for the Link failure detected by the Openflow plugin.