Operating System: All
Platform: All

It seems to me that there were other applications which installed flow entries during the tests.

The following page shows flow entries during failed test case.
https://jenkins.opendaylight.org/releng/view/vtn/job/vtn-csit-1node-manager-all-beryllium/81/robot/report/log.html#s1-s1-s3-t19-k1-k20-k1-k1-k3-k1-k1
However, it seems to me that these flow entries are different from what VTN Manager installs.
I guess other application installed these flow entries.
I guess that is related to the failure (VTN-121).

For your information, the following page shows flow entries while this test case worked well.
https://jenkins.opendaylight.org/releng/view/vtn/job/vtn-csit-1node-manager-all-beryllium/82/robot/report/log.html#s1-s1-s3-t19-k1-k1-k1-k1-k2

Reading the karaf.log on the CSIT, I've noticed that SFC OFL2 Renderer and NetVirt was installed. These are applications which install flow entries into OpenFlow switches.

And, I've found the following patch which, I guess, would fix the issue.
https://git.opendaylight.org/gerrit/#/c/34780/

Let's wait for the merge of the patch, and see if the same failure happens after the merge.

The error have not occurred in the CSIT for beryllium since the following patch is merged into the beryllium branch of the distribution group.
https://git.opendaylight.org/gerrit/#/c/34780/

However, Similar failures happened in the CSIT for the master branch.
https://jenkins.opendaylight.org/releng/job/vtn-csit-1node-manager-only-boron/133/robot/report/log.html#s1-s1-s3

FYI, I'm waiting the following patch is merged.
https://git.opendaylight.org/gerrit/#/c/35352/
(The master branch of the integration/distribution.git.)

While i test in sandbox with latest Beryllium code. I find two test case failed in sandbox.

There was failure in flowfilter and below is the link.
https://jenkins.opendaylight.org/sandbox/job/vtn-csit-1node-manager-all-beryllium/2/robot/report/log.html#s1-s1-s3-t32

https://jenkins.opendaylight.org/sandbox/job/vtn-csit-1node-manager-all-beryllium/2/robot/report/log.html#s1-s2-s4-t23

Analysis of the CSIT flow-filter test case failures

The CSIT flow filter test case was failing in the CSIT setup for inet-src-action/inet-dst-action.

Test scenarios Failed
• VTN Flow filter
• VBridge flow filter.

Test scenarios Passed
• VBridge Interface flow filter.

Order of Test scenarios execution
• VTN Flow filter
• vBridge Flow filter
• vBridge Interface flow filter.

Initial analysis of the test case
• Incorrect inputs values were provided to the inet-ser/inet-dst parameters
• The ping was expected to be success as the input parameters were incorrect.
• Dump flow analysis verification was incorrect as the parameters were incorrect.

Changes done to the test case
• Input parameters were modified to expected values for inet-ser/inet-dst parameters
• The ping verification was modified for failure as the src/dst ip address were modified.
• Dump flow analysis verification was modified to verify the correct set action in the edge port switch.

Topology used
h1,h2 <--> s2 <-> s1 <-> s3 <--> h3,h4

Following are the steps followed in the CSIT test script.

1. Create Tenant1
2. Create vBridge1
3. Create interfaces(if1 and if2) in vBridge1
4. Create portmapping for if1 and if2.
5. Ping h1 and h3
6. Verify Ping success
7. Create vBridge2,
8. Create interfaces(if3and if4) in Vbridge 2,
9. Create portmapping for if3 and if4.
10. Ping h3 and h4
11. Verify ping success
12. Add flow condition to match h1, h3 ip address
13. Add VTN flowfilter to set flow action (vtn-set-inet-src-action/ vtn-set-inet-dst-action)
14. Ping h1 and h3
15. Verify ping failure
16. Read the flow entries in mininet switch s3 to verify actions=mod_nw_src is set or not.
17. Add VBR flowfilter to set flow action (vtn-set-inet-src-action/ vtn-set-inet-dst-action)
18. Ping h1 and h3
19. Verify ping failure
20. Read the flow entries in mininet switch s3 to verify actions=mod_nw_src is set or not.
21. Add VBRIf flowfilter in If1 to set flow action (vtn-set-inet-src-action/ vtn-set-inet-dst-action)
22. Ping h1 and h3
23. Verify ping failure
24. Read the flow entries in mininet switch s3 to verify actions=mod_nw_src is set or not.
25. Delete flow condition
26. Delete vtn1

In the above steps step 16 and step 20 failed but step 24 is success.
After the above changes were done, It was observed that steps 16 and step 20 were still failing as before. The issue was the correct flows were not available in the switch when the flows were dumped in mininet switch. So the flow verification test case failed.
To understand the issues, Karaf log was analyzed for VTN Manager module while the above steps were executed.
Karaf log file name - all_flowfilter.log

VTN flow filter - Test steps 13 – 16.

a) Flow entry install request is issued in line 1521 at time stamp 2016-03-04 10:15:30,965
b) Flow entries remove request has be issued in line 1531 at time stamp 2016-03-04 10:15:31,034

VBR Flow filter – Test steps 17 – 20
a) Flow entry install request is issued in line 1905 at time stamp 2016-03-04 10:15:53,201
b) Flow entries remove request has be issued in line 1934 at time stamp 2016-03-04 10:15:56,781

In both the above cases its observed that the flow entries are removed after the below log message
Ignore packet from internal node connector: openflow:2:3

VBR Flow filter – Test steps 21 – 24
a) Flow entry install request is issued in line 2112 at time stamp 2016-03-04 10:16:16,010
b) Flow entries remove request has be issued in line 1934 at time stamp 2016-03-04 10:16:28,130

In this case the flow verification is success as the flows are removed only during the removal of the vtn configuration during step 25, 26.

From our analysis, the flows for VTN Flow filter and VBridge flowfilter are removed in VTN Manager before the test can verify the flows.

Tried to verify this behavior by manually creating VTN, VBridge and VBridge IF flow filter. But not able to reproduce the same behavior.

Manual test – vtn flow filter
Manually Tested creating vtn flowfilter with the above scenario.
Configured flow filters, and send packets which match the flow filters.
After that, checked if flow entries for the flow filters are correctly created in mininet.
As expected Flow entry created in Switch s3.

In manualflow.log

1) Created VTN, VBR, IF1 and IF2, portmapping IF1 and IF2.
2) Added Flowconditions and created VTN flowfilter with actions inetdrc and inetdst.
3) As expected Ping fails between hosts h1 and h3.
4) Checked dumpflows flow actions listed in mininet dumpflows.
5) In manualflow.log Line No: :2112, Flow entries has been created at time stamp 2016-03-07 04:42:27,685
6) Flow entries is not deleted unless we manually remove flows.

Request VTN Manager teams support to analyze the logs and inform if there is any issue in the CSIT test procedure.

Attached log file ff_karaf.log

Attachment ff_karaf.log has been added with description: Karaf log enabled TRACE

The problem still happened in recent builds.

For example:
https://jenkins.opendaylight.org/releng/view/vtn/job/vtn-csit-1node-manager-only-beryllium/216/
(Mar 6, 2016)

The following CSIT for Beryllium to verify flowentries are working fine in below CSIT Beryllium jobs.
https://jenkins.opendaylight.org/releng/view/vtn/job/vtn-csit-1node-manager-only-beryllium/
https://jenkins.opendaylight.org/releng/view/vtn/job/vtn-csit-1node-manager-all-beryllium/

From 11/03/2016 we have checked the above CSIT jobs and there is no failures occurred which is related to Verify flowentries.

The issue is fixed after the following patch is merged.
https://git.opendaylight.org/gerrit/#/c/36039/