Operating System: All
Platform: All

I guess this is a bug of the CSIT.
Therefore I think the CSIT needs to be modified.

I've found out that the CSIT uses the following data store to get data flows from VTN Manager in the failed test case.

restconf/operational/vtn-flow-impl:vtn-flows/

However, this data model (vtn-flow-impl) should not be used in the CSIT, since this data model is an internal model in VTN Manager and not assumed to be used by applications.
This is explicitly mentioned in vtn-flow-impl.yang file.

https://git.opendaylight.org/gerrit/gitweb?p=vtn.git;a=blob;f=manager/implementation/src/main/yang/vtn-flow-impl.yang;h=aeb318fd209a843f6106a515352dfe88d3c2179c;hb=refs/heads/stable/beryllium

66 description
67 "The internal module for entities implementing the VTN data flow.
68 Application other than VTN Manager must not use models defined in
69 this module.";

Instead of that, please use public data stores and RPCs provided by VTN Manager.
The explanation for all public models are available at the Jenkins server.
https://jenkins.opendaylight.org/releng/view/vtn/job/vtn-merge-beryllium/lastSuccessfulBuild/artifact/manager/model/target/site/models/

First, I think the failed test case should use the get-data-flow RPC provided by VTN Manager to verify if target data flows exist.
https://jenkins.opendaylight.org/releng/view/vtn/job/vtn-merge-beryllium/lastSuccessfulBuild/artifact/manager/model/target/site/models/vtn-flow.html#get-data-flow

Second, since anyway VTN Manager needs some time to be ready to return the target data flow information, I think the CSIT needs to add some sleep time before the verification or needs to repeat the verification a few times.

Third, I'm not sure but specifying the "UPDATESTATS" into the "mode" parameter of the get-data-flow RPC might solve the timing issue, since the get-data-flow PRC in the "UPDATESTATS" mode makes requests to OpenFlow switch to get up-to-date information.

I've found out that the same issue (VTN-120) happened in the build #129 of the vtn-csit-1node-manager-only-beryllium job.

https://jenkins.opendaylight.org/releng/view/vtn/job/vtn-csit-1node-manager-only-beryllium/129/robot/report/log.html#s1-s1-s2-t18

Modifed VTN Manager CSIT Keyword file. To use get-data-flow RPC provided by VTN manager.

Pushed patch and merged in to integration/test,
https://git.opendaylight.org/gerrit/#/c/34735

Tested in Sandbox now verify dataflow which uses get-data-flow RPC works fine.
https://jenkins.opendaylight.org/sandbox/job/vtn-csit-1node-manager-only-beryllium/19/console

After the patch is merged to git, the below job failed once in the below

https://jenkins.opendaylight.org/releng/view/vtn/job/vtn-csit-1node-manager-all-beryllium/88/robot/report/log.html#s1-s2-s3-t11

Before pushing the patch to git, i have tested 5 times in sandbox but i didn't see this error.
The failure see only once in all-beryllium job. Other jobs are working fine without any issue.

I think, vtn manager need some time to return dataflow information. We can add some "Wait Until Keyword"(sleep) for Verify dataflow test cases.

CSIT for Beryllium VTN Manager failed to verify data flows.

Topology used
h1,h2 <--> s2 <-> s1 <-> s3 <--> h3,h4

Following are the steps followed in the CSIT test script.

1) Create Tenant1
2) Create vBridge1
3) Create interfaces if1 and if2
4) Create Portmapping for interfaces if1 and if2
5) Ping h1 and h3.  Ping success
6) Verify Dataflows.  Success. Here verifying dataflow output for “Tenant1, vBridge1 , if1 and PORTMAPPED”
7) Create vBridge2
8) Create interfaces if3 and if4
9) Create Portmapping for interfaces if3 and if4
10) Ping h2 and h4.  Ping success
11) Verify Dataflows.  failure. Here verifying dataflow output for “Tenant1, vBridge2 , if3 and PORTMAPPED”

From the Karaf Logs it is observed that when the get vtn data flow request is made for
• vbridge 1 - The relevant stats request are seen.
• vBridge 2 – The relevant stats request are not seen.

One more observation - Sometimes Verify dataflow for vbridge2 is successful after some retries. In such a case the status requests are seen in the logs.

Request VTN Manager teams support to analyze the logs and inform if there is any issue in the CSIT test procedure.

Log files are attached
vtndf_karaf.log
vtndf_karaf.log.1

Attachment vtndf_karaf.log has been added with description: Karaf log enabled TRACE

Attachment vtndf_karaf_log.zip has been added with description: Karaf log enabled TRACE

(In reply to Karthik Sivasamy from comment #6)
> CSIT for Beryllium VTN Manager failed to verify data flows.
>
> Topology used
> h1,h2 <--> s2 <-> s1 <-> s3 <--> h3,h4
>
> Following are the steps followed in the CSIT test script.
>
> 1) Create Tenant1
> 2) Create vBridge1
> 3) Create interfaces if1 and if2
> 4) Create Portmapping for interfaces if1 and if2
> 5) Ping h1 and h3.  Ping success
> 6) Verify Dataflows.  Success. Here verifying dataflow output for
> “Tenant1, vBridge1 , if1 and PORTMAPPED”
> 7) Create vBridge2
> 8) Create interfaces if3 and if4
> 9) Create Portmapping for interfaces if3 and if4
> 10) Ping h2 and h4.  Ping success
> 11) Verify Dataflows.  failure. Here verifying dataflow output for
> “Tenant1, vBridge2 , if3 and PORTMAPPED”
>
> From the Karaf Logs it is observed that when the get vtn data flow request
> is made for
> • vbridge 1 - The relevant stats request are seen.
> • vBridge 2 – The relevant stats request are not seen.
>
> One more observation - Sometimes Verify dataflow for vbridge2 is successful
> after some retries. In such a case the status requests are seen in the logs.
>
> Request VTN Manager teams support to analyze the logs and inform if there is
> any issue in the CSIT test procedure.
>
> Log files are attached
> vtndf_karaf.log
> vtndf_karaf.log.1

Ignore the above log files. attached the zip file vtndf_karaf_log.zip with includes both the files

Exceptions in karaf.log
016-03-07 06:53:27,969 | ERROR | er Task Thread-0 | StatsReaderService | 188 - org.opendaylight.vtn.manager.implementation - 0.4.0.Beryllium | get-flow-stats: Caught an exception: canceled=false, input=GetFlowStatisticsFromFlowTableInput [_cookie=FlowCookie [_value=9175521290813964288], _cookieMask=FlowCookie [_value=18446462598732840960], _node=NodeRef [_value=KeyedInstanceIdentifier

], _tableId=0, augmentation=[]]
java.util.concurrent.ExecutionException: org.opendaylight.controller.md.sal.dom.api.DOMRpcImplementationNotAvailableException: No local or remote implementation available for rpc AbsoluteSchemaPath

at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299)[64:com.google.guava:18.0.0]
at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:272)[64:com.google.guava:18.0.0]
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:96)[64:com.google.guava:18.0.0]
at org.opendaylight.vtn.manager.internal.util.rpc.RpcInvocation.getResult(RpcInvocation.java:82)[188:org.opendaylight.vtn.manager.implementation:0.4.0.Beryllium]
at org.opendaylight.vtn.manager.internal.util.flow.GetFlowStatsRpc.getTransactionId(GetFlowStatsRpc.java:56)[188:org.opendaylight.vtn.manager.implementation:0.4.0.Beryllium]
at org.opendaylight.vtn.manager.internal.flow.stats.StatsReaderService.startTransaction(StatsReaderService.java:546)[188:org.opendaylight.vtn.manager.implementation:0.4.0.Beryllium]
at org.opendaylight.vtn.manager.internal.flow.stats.StatsReaderService.access$400(StatsReaderService.java:61)[188:org.opendaylight.vtn.manager.implementation:0.4.0.Beryllium]
at org.opendaylight.vtn.manager.internal.flow.stats.StatsReaderService$RpcTask.run(StatsReaderService.java:234)[188:org.opendaylight.vtn.manager.implementation:0.4.0.Beryllium]
at org.opendaylight.vtn.manager.internal.util.concurrent.VTNThreadPool$WorkerThread.run(VTNThreadPool.java:400)[188:org.opendaylight.vtn.manager.implementation:0.4.0.Beryllium]
Caused by: org.opendaylight.controller.md.sal.dom.api.DOMRpcImplementationNotAvailableException: No local or remote implementation available for rpc AbsoluteSchemaPath

at org.opendaylight.controller.remote.rpc.RemoteRpcImplementation$1.onComplete(RemoteRpcImplementation.java:65)[163:org.opendaylight.controller.sal-remoterpc-connector:1.3.0.Beryllium]
at org.opendaylight.controller.remote.rpc.RemoteRpcImplementation$1.onComplete(RemoteRpcImplementation.java:56)[163:org.opendaylight.controller.sal-remoterpc-connector:1.3.0.Beryllium]
at akka.dispatch.OnComplete.internal(Future.scala:248)[149:com.typesafe.akka.actor:2.3.14]
at akka.dispatch.OnComplete.internal(Future.scala:245)[149:com.typesafe.akka.actor:2.3.14]
at akka.dispatch.japi$CallbackBridge.apply(Future.scala:175)[149:com.typesafe.akka.actor:2.3.14]
at akka.dispatch.japi$CallbackBridge.apply(Future.scala:172)[149:com.typesafe.akka.actor:2.3.14]
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)[146:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c]
at scala.concurrent.impl.ExecutionContextImpl$AdaptedForkJoinTask.exec(ExecutionContextImpl.scala:121)[146:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c]
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)[146:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c]
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)[146:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c]
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)[146:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c]
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)[146:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c]
2016-03-07 06:53:27,975 | ERROR | er Task Thread-0 | StatsReaderService | 188 - org.opendaylight.vtn.manager.implementation - 0.4.0.Beryllium | Failed to issue flow stats RPC: get-flow-stats
org.opendaylight.vtn.manager.VTNException: No local or remote implementation available for rpc AbsoluteSchemaPath

at org.opendaylight.vtn.manager.internal.util.concurrent.AbstractVTNFuture.getException(AbstractVTNFuture.java:75)[188:org.opendaylight.vtn.manager.implementation:0.4.0.Beryllium]
at org.opendaylight.vtn.manager.internal.util.rpc.RpcInvocation.getResult(RpcInvocation.java:89)[188:org.opendaylight.vtn.manager.implementation:0.4.0.Beryllium]
at org.opendaylight.vtn.manager.internal.util.flow.GetFlowStatsRpc.getTransactionId(GetFlowStatsRpc.java:56)[188:org.opendaylight.vtn.manager.implementation:0.4.0.Beryllium]
at org.opendaylight.vtn.manager.internal.flow.stats.StatsReaderService.startTransaction(StatsReaderService.java:546)[188:org.opendaylight.vtn.manager.implementation:0.4.0.Beryllium]
at org.opendaylight.vtn.manager.internal.flow.stats.StatsReaderService.access$400(StatsReaderService.java:61)[188:org.opendaylight.vtn.manager.implementation:0.4.0.Beryllium]
at org.opendaylight.vtn.manager.internal.flow.stats.StatsReaderService$RpcTask.run(StatsReaderService.java:234)[188:org.opendaylight.vtn.manager.implementation:0.4.0.Beryllium]
at org.opendaylight.vtn.manager.internal.util.concurrent.VTNThreadPool$WorkerThread.run(VTNThreadPool.java:400)[188:org.opendaylight.vtn.manager.implementation:0.4.0.Beryllium]
Caused by: org.opendaylight.controller.md.sal.dom.api.DOMRpcImplementationNotAvailableException: No local or remote implementation available for rpc AbsoluteSchemaPath

at org.opendaylight.controller.remote.rpc.RemoteRpcImplementation$1.onComplete(RemoteRpcImplementation.java:65)[163:org.opendaylight.controller.sal-remoterpc-connector:1.3.0.Beryllium]
at org.opendaylight.controller.remote.rpc.RemoteRpcImplementation$1.onComplete(RemoteRpcImplementation.java:56)[163:org.opendaylight.controller.sal-remoterpc-connector:1.3.0.Beryllium]
at akka.dispatch.OnComplete.internal(Future.scala:248)[149:com.typesafe.akka.actor:2.3.14]
at akka.dispatch.OnComplete.internal(Future.scala:245)[149:com.typesafe.akka.actor:2.3.14]
at akka.dispatch.japi$CallbackBridge.apply(Future.scala:175)[149:com.typesafe.akka.actor:2.3.14]
at akka.dispatch.japi$CallbackBridge.apply(Future.scala:172)[149:com.typesafe.akka.actor:2.3.14]
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)[146:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c]
at scala.concurrent.impl.ExecutionContextImpl$AdaptedForkJoinTask.exec(ExecutionContextImpl.scala:121)[146:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c]
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)[146:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c]
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)[146:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c]
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)[146:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c]
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)[146:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c]
2016-03-07 06:53:27,976 | WARN | er Task Thread-0 | ClearNodeFlows

(In reply to Venkatrangan Govindarajan from comment #10)
> Exceptions in karaf.log
> 016-03-07 06:53:27,969 | ERROR | er Task Thread-0 | StatsReaderService
> | 188 - org.opendaylight.vtn.manager.implementation - 0.4.0.Beryllium |
> get-flow-stats: Caught an exception: canceled=false,

I've filed that ERROR message as OPNFLWPLUG-605.
https://bugs.opendaylight.org/show_bug.cgi?id=5090

And, I don't think that ERROR message is related to the problem of the VTN-120.

Thanks for clarifying, Just looking at the exception, felt like the stats for some switch is failing hence added it here. Will look further and update

Since VTN Manager needs some time to be ready to return data flow information after it receives new packets, I think the CSIT needs to add some sleep time before the verification or needs to repeat the verification a few times.

I think the patch for VTN-121 also fixes the VTN-120 since the root cause of these issues could be the same.
https://git.opendaylight.org/gerrit/#/c/36039/

The following CSIT for Beryllium to verify Dataflows are working fine.
https://jenkins.opendaylight.org/releng/view/vtn/job/vtn-csit-1node-manager-only-beryllium/
https://jenkins.opendaylight.org/releng/view/vtn/job/vtn-csit-1node-manager-all-beryllium/

Since after the following patch was merged,
https://git.opendaylight.org/gerrit/#/c/36039/

To fix the CSIT test failure, we have used the get-data-flow RPC provided by VTN Manager to verify if target data flows exist.

https://jenkins.opendaylight.org/releng/view/vtn/job/vtn-merge-beryllium/lastSuccessfulBuild/artifact/manager/model/target/site/models/vtn-flow.html#get-data-flow

After fixing the CSIT we got some other issue, that is verify dataflow fails and the issue is fixed with the following patch.
https://git.opendaylight.org/gerrit/#/c/36039/

From 11/03/2016, since after the above patch was merged, we dont see any issue related to Verify Dataflow tests in the follwing CSIT Beryllium jobs.

https://jenkins.opendaylight.org/releng/view/vtn/job/vtn-csit-1node-manager-only-beryllium/
https://jenkins.opendaylight.org/releng/view/vtn/job/vtn-csit-1node-manager-all-beryllium/