https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/CONTROLLER-1550 controller <p>Basic flow installation is not working in boron. Refer below regression report.</p> <p><a href="https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-3node-periodic-bulkomatic-clustering-daily-only-boron/97/robot/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-3node-periodic-bulkomatic-clustering-daily-only-boron/97/robot/</a></p> <p>Steps where it is failing:</p> <p> openflowplugin-bulkomatic-clustering-daily.txt.Cluster HA Data Recovery BulkFlow 2Node Cluster.Verify Flows In Switch Before Cluster Restart<br/> Tags: <br/> Message: <br/> Keyword 'MininetKeywords.Mininet Sync Status' failed after retrying for 1 minute 40 seconds. The last error was: '*** s1 ------------------------------------------------------------------------<br/> OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=0<br/> mininet&gt;' contains 'flow_count=1000' 0 times, not 1 time.<br/> no 0:01:41.536 2<br/> openflowplugin-bulkomatic-clustering-daily.txt.Cluster HA Data Recovery BulkFlow 2Node Cluster.Verify Flows In Switch Before Follower Restart<br/> Tags: <br/> Message: <br/> Keyword 'MininetKeywords.Mininet Sync Status' failed after retrying for 1 minute 40 seconds. The last error was: '*** s1 ------------------------------------------------------------------------<br/> OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=0<br/> mininet&gt;' contains 'flow_count=1000' 0 times, not 1 time.</p> <p>Scenario Execution Step:<br/> ==========================</p> <p>1. Get Inventory Follower and Leader information</p> <p>2. Start Mininet Connect To Follower Node1</p> <p>3. Add Bulk Flow ( 1000 flows )From Follower Node1</p> <p>4. Get Bulk Flows and Verify In Cluster config DS all flows are installed</p> <p>5. Verify Flows In Switch , The test is failing in this step. no flows seen in OVS switch.</p> <p>Attached the follower node1 karaf logs</p> <p>Operating System: All<br/> Platform: All</p> CONTROLLER-1550

Perf test 100K flow installation does not always work in Cluster

Bug Medium Resolved Done Luis Gomez Sanjib Mohapatra Tue, 20 Sep 2016 10:54:17 +0000 Tue, 25 Jul 2023 08:24:13 +0000 Fri, 5 Oct 2018 15:19:39 +0000 clustering 0 9 <p>Attachment karaf.rar has been added with description: follower node1 karaf logs</p> <p>Isn't this duplicate of this one? <a href="https://bugs.opendaylight.org/show_bug.cgi?id=6552" class="external-link" target="_blank" rel="nofollow noopener">https://bugs.opendaylight.org/show_bug.cgi?id=6552</a></p> <p>I do not think there is regression here, the failures are in the cluster performance test that we have to separate in its own job and adjust so that it passes.</p> <p>Reducing severity and change title to describe exactly what is going on.</p> <p>The test is sporadically failing with just 32 switches and 2K flows/switch (64K flows):</p> <p><a href="https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-3node-periodic-bulkomatic-clustering-perf-daily-only-carbon/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-3node-periodic-bulkomatic-clustering-perf-daily-only-carbon/</a></p> <p>Therefore rising to critical for now.</p> <p>I am adding more memory to see if it helps:</p> <p><a href="https://git.opendaylight.org/gerrit/#/c/46644/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/46644/</a></p> <p>Luis,</p> <p>Did adding more memory to the test (as per your last comment) help?</p> <p>Abhijit</p> <p>No it did not <img class="emoticon" src="https://jira.opendaylight.org/images/icons/emoticons/sad.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>For this particular bug I have action point of collecting cluster releated ERRORs in karaf log and open bugs to cluster project.</p> <p>It seems that Boron is passing fine but not Carbon. For the latest I opened this bug in controller project to better understand the ERRORs:</p> <p><a href="https://bugs.opendaylight.org/show_bug.cgi?id=7901" class="external-link" target="_blank" rel="nofollow noopener">https://bugs.opendaylight.org/show_bug.cgi?id=7901</a></p> <p>Anil Vishnoi looking at this. The patch for this is failing netvirt test cases - and was reverted by <a href="https://git.opendaylight.org/gerrit/#/c/53532/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/53532/</a>.</p> <p>My current observation is that stats collection for 100K flows work most of the times in Boron:</p> <p><a href="https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-3node-periodic-bulkomatic-clustering-daily-only-boron" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-3node-periodic-bulkomatic-clustering-daily-only-boron</a></p> <p>But never works in Carbon:</p> <p><a href="https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-3node-periodic-bulkomatic-clustering-daily-only-carbon/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-3node-periodic-bulkomatic-clustering-daily-only-carbon/</a></p> <p>So we may be facing a perf regression in Carbon.</p> <p>BR/Luis</p> <p>Luis, the test are showing </p> <p>Variable '${temp_json_config_get}' not found.</p> <p>It is some issue in test itself ?</p> <p>I actually posted the wrong links, these are the goods:</p> <p>Boron:</p> <p><a href="https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-3node-periodic-bulkomatic-clustering-perf-daily-only-boron" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-3node-periodic-bulkomatic-clustering-perf-daily-only-boron</a></p> <p>Carbon:</p> <p><a href="https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-3node-periodic-bulkomatic-clustering-perf-daily-only-carbon/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-3node-periodic-bulkomatic-clustering-perf-daily-only-carbon/</a></p> <p>BR/Luis</p> <p>Hi Luis,</p> <p>When I tried to reproduce this locally, I have received following warning:<br/> (after all flows added to the config)</p> <p>15159:2017-05-29 14:40:34,956 | WARN | entLoopGroup-5-2 | OperationLimiter | 207 - org.opendaylight.controller.sal-distributed-datastore - 1.5.1.SNAPSHOT | Failed to acquire operation permit for transaction member-1-datastore-operational-fe-0-chn-12-txn-734-0</p> <p>It does not appear in Jenkins logs. Have you seen it already somewhere?</p> <p>It is happening already when I try to add 1000 flows on 10 switches.</p> <p>Flows are getting to the config and also to the switch.</p> <p>But then mentioned WARN appear and later some switches are disconnected.</p> <p>I have connected only one controller based on Jenkins test (one of the Followers), and it was timed out.</p> <p>The Master was lost. And since no other controller was connected, we are finished...</p> <p>I am wondering, if this can happen on Jenkins also, since logs don't seem to be same.</p> <p>Thanks,<br/> Miro</p> <p>This is because in scalability/perf tests we use log:set ERROR. I will file a patch to set WARN on this test.</p> <p><a href="https://git.opendaylight.org/gerrit/#/c/58016" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/58016</a></p> <p>Hi Luis,</p> <p>Great. Thanks a lot.</p> <p>One run was completed already, but there are no logs.</p> <p><a href="https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-3node-periodic-bulkomatic-clustering-perf-daily-only-carbon/259/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-3node-periodic-bulkomatic-clustering-perf-daily-only-carbon/259/</a></p> <p>I believe it happen just occasionally, since there are also other builds without logs. Right?</p> <p>Hope next run logs should be there.</p> <p>Thanks,<br/> Miro</p> <p>Right, next build worked: <a href="https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-3node-periodic-bulkomatic-clustering-perf-daily-only-carbon/260/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-3node-periodic-bulkomatic-clustering-perf-daily-only-carbon/260/</a></p> <p>Hello Luis,</p> <p>We need to check results of this test:</p> <p><a href="https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-3node-periodic-bulkomatic-clustering-perf-daily-only-nitrogen/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-3node-periodic-bulkomatic-clustering-perf-daily-only-nitrogen/</a></p> <p>But it looks like it is not working properly. In the graph it is showing pass green, tests are red and logs are not there.</p> <p>Could you please check what can be wrong?</p> <p>Thanks a lot,<br/> Miro</p> <p>There is already a thread on this, the issue is impacting more projects (not just ofp):</p> <p><a href="https://lists.opendaylight.org/pipermail/integration-dev/2017-June/009906.html" class="external-link" target="_blank" rel="nofollow noopener">https://lists.opendaylight.org/pipermail/integration-dev/2017-June/009906.html</a></p> <p>I will let you know when it gets fixed.</p> <p>Hi Miroslav, Nitrogen pass some times now:</p> <p><a href="https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-3node-periodic-bulkomatic-clustering-perf-daily-only-nitrogen/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-3node-periodic-bulkomatic-clustering-perf-daily-only-nitrogen/</a></p> <p>But Carbon fails all the time:</p> <p><a href="https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-3node-periodic-bulkomatic-clustering-perf-daily-only-carbon/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-3node-periodic-bulkomatic-clustering-perf-daily-only-carbon/</a></p> <p>Issue sems to be flow stats collection.</p> <p>So it looks like on nitrogen this is already working, and on carbon it will be working after this cherry-pick will be merged: <a href="https://git.opendaylight.org/gerrit/#/c/60196/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/60196/</a></p> <p>Cherry-pick was merged and looks like it is working on carbon too now (<a href="https://jenkins.opendaylight.org/releng/job/openflowplugin-csit-3node-periodic-gate-bulkomatic-clustering-perf-daily-only-carbon/98/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/job/openflowplugin-csit-3node-periodic-gate-bulkomatic-clustering-perf-daily-only-carbon/98/</a>) so closing this bug.</p> <p>Yes that is fine, we have all branches behaving the same, but do we want to leave this open to track the issue of sporadic failure in this test? also looking at karaf.log when it fails, it looks like we are hitting cluster issues:</p> <p>2017-07-17 18:04:34,185 | WARN | tp1066220241-427 | BrokerFacade | 276 - org.opendaylight.netconf.sal-rest-connector - 1.5.2.SNAPSHOT | Error reading /(urn:opendaylight:inventory?revision=2013-08-19)nodes from datastore OPERATIONAL<br/> ReadFailedException</p> {message=Error checking ReadData for path /(urn:opendaylight:inventory?revision=2013-08-19)nodes, errorList=[RpcError [message=Error checking ReadData for path /(urn:opendaylight:inventory?revision=2013-08-19)nodes, severity=ERROR, errorType=APPLICATION, tag=operation-failed, applicationTag=null, info=null, cause=akka.pattern.AskTimeoutException: Ask timed out on [ActorSelection[Anchor(akka.tcp://opendaylight-cluster-data@10.29.12.99:2550/), Path(/user/shardmanager-operational/member-3-shard-inventory-operational/shard-inventory-member-1:datastore-operational@0:8_1309#1833809227)]] after [5000 ms]. Sender[null] sent message of type "org.opendaylight.controller.cluster.datastore.messages.ReadData".]]} <p>Just realized there is a regression if we compare to Boron:</p> <p><a href="https://jenkins.opendaylight.org/releng/job/openflowplugin-csit-3node-periodic-bulkomatic-clustering-perf-daily-only-boron/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/job/openflowplugin-csit-3node-periodic-bulkomatic-clustering-perf-daily-only-boron/</a></p> <p>Or even nitrogen:</p> <p><a href="https://jenkins.opendaylight.org/releng/job/openflowplugin-csit-3node-periodic-bulkomatic-clustering-perf-daily-only-nitrogen/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/job/openflowplugin-csit-3node-periodic-bulkomatic-clustering-perf-daily-only-nitrogen/</a></p> <p>So reopening the bug for this:</p> <p><a href="https://jenkins.opendaylight.org/releng/job/openflowplugin-csit-3node-periodic-bulkomatic-clustering-perf-daily-only-carbon/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/job/openflowplugin-csit-3node-periodic-bulkomatic-clustering-perf-daily-only-carbon/</a></p> <p>Luis</p> <p>Looking at a failed job log, I think this is cluster issue so moving bug to the right project:</p> <p>2017-09-17 15:53:44,930 | WARN | tp1225015068-434 | BrokerFacade | 276 - org.opendaylight.netconf.sal-rest-connector - 1.5.2.SNAPSHOT | Error reading /(urn:opendaylight:inventory?revision=2013-08-19)nodes from datastore OPERATIONAL<br/> ReadFailedException</p> {message=Error checking ReadData for path /(urn:opendaylight:inventory?revision=2013-08-19)nodes, errorList=[RpcError [message=Error checking ReadData for path /(urn:opendaylight:inventory?revision=2013-08-19)nodes, severity=ERROR, errorType=APPLICATION, tag=operation-failed, applicationTag=null, info=null, cause=akka.pattern.AskTimeoutException: Ask timed out on [ActorSelection[Anchor(akka.tcp://opendaylight-cluster-data@10.29.14.94:2550/), Path(/user/shardmanager-operational/member-3-shard-inventory-operational/shard-inventory-member-2:datastore-operational@0:6_870#-1165819976)]] after [5000 ms]. Sender[null] sent message of type "org.opendaylight.controller.cluster.datastore.messages.ReadData".]]} <p> at org.opendaylight.controller.cluster.datastore.RemoteTransactionContext$1.onComplete(RemoteTransactionContext.java:184)<span class="error">&#91;196:org.opendaylight.controller.sal-distributed-datastore:1.5.2.SNAPSHOT&#93;</span><br/> at akka.dispatch.OnComplete.internal(Future.scala:258)<span class="error">&#91;170:com.typesafe.akka.actor:2.4.18&#93;</span><br/> at akka.dispatch.OnComplete.internal(Future.scala:256)<span class="error">&#91;170:com.typesafe.akka.actor:2.4.18&#93;</span><br/> at akka.dispatch.japi$CallbackBridge.apply(Future.scala:186)<span class="error">&#91;170:com.typesafe.akka.actor:2.4.18&#93;</span><br/> at akka.dispatch.japi$CallbackBridge.apply(Future.scala:183)<span class="error">&#91;170:com.typesafe.akka.actor:2.4.18&#93;</span><br/> at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:36)<span class="error">&#91;166:org.scala-lang.scala-library:2.11.11.v20170413-090219-8a413ba7cc&#93;</span><br/> at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)<span class="error">&#91;170:com.typesafe.akka.actor:2.4.18&#93;</span><br/> at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:91)<span class="error">&#91;170:com.typesafe.akka.actor:2.4.18&#93;</span><br/> at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)<span class="error">&#91;170:com.typesafe.akka.actor:2.4.18&#93;</span><br/> at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)<span class="error">&#91;170:com.typesafe.akka.actor:2.4.18&#93;</span><br/> at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)<span class="error">&#91;166:org.scala-lang.scala-library:2.11.11.v20170413-090219-8a413ba7cc&#93;</span><br/> at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:90)<span class="error">&#91;170:com.typesafe.akka.actor:2.4.18&#93;</span><br/> at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:39)<span class="error">&#91;170:com.typesafe.akka.actor:2.4.18&#93;</span><br/> at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:415)<span class="error">&#91;170:com.typesafe.akka.actor:2.4.18&#93;</span><br/> at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)<span class="error">&#91;166:org.scala-lang.scala-library:2.11.11.v20170413-090219-8a413ba7cc&#93;</span><br/> at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)<span class="error">&#91;166:org.scala-lang.scala-library:2.11.11.v20170413-090219-8a413ba7cc&#93;</span><br/> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)<span class="error">&#91;166:org.scala-lang.scala-library:2.11.11.v20170413-090219-8a413ba7cc&#93;</span><br/> at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)<span class="error">&#91;166:org.scala-lang.scala-library:2.11.11.v20170413-090219-8a413ba7cc&#93;</span><br/> Caused by: akka.pattern.AskTimeoutException: Ask timed out on [ActorSelection<span class="error">&#91;Anchor(akka.tcp://opendaylight-cluster-data@10.29.14.94:2550/), Path(/user/shardmanager-operational/member-3-shard-inventory-operational/shard-inventory-member-2:datastore-operational@0:6_870#-1165819976)&#93;</span>] after <span class="error">&#91;5000 ms&#93;</span>. Sender<span class="error">&#91;null&#93;</span> sent message of type "org.opendaylight.controller.cluster.datastore.messages.ReadData".<br/> at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:604)<span class="error">&#91;170:com.typesafe.akka.actor:2.4.18&#93;</span><br/> at akka.actor.Scheduler$$anon$4.run(Scheduler.scala:126)<span class="error">&#91;170:com.typesafe.akka.actor:2.4.18&#93;</span><br/> at scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:601)<span class="error">&#91;166:org.scala-lang.scala-library:2.11.11.v20170413-090219-8a413ba7cc&#93;</span><br/> at scala.concurrent.BatchingExecutor$class.execute(BatchingExecutor.scala:109)<span class="error">&#91;166:org.scala-lang.scala-library:2.11.11.v20170413-090219-8a413ba7cc&#93;</span><br/> at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:599)<span class="error">&#91;166:org.scala-lang.scala-library:2.11.11.v20170413-090219-8a413ba7cc&#93;</span><br/> at akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:329)<span class="error">&#91;170:com.typesafe.akka.actor:2.4.18&#93;</span><br/> at akka.actor.LightArrayRevolverScheduler$$anon$4.executeBucket$1(LightArrayRevolverScheduler.scala:280)<span class="error">&#91;170:com.typesafe.akka.actor:2.4.18&#93;</span><br/> at akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:284)<span class="error">&#91;170:com.typesafe.akka.actor:2.4.18&#93;</span><br/> at akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:236)<span class="error">&#91;170:com.typesafe.akka.actor:2.4.18&#93;</span><br/> at java.lang.Thread.run(Thread.java:748)<span class="error">&#91;:1.8.0_141&#93;</span><br/> 2017-09-17 15:53:47,162 | WARN | lt-dispatcher-47 | ClusterCoreDaemon | 171 - com.typesafe.akka.slf4j - 2.4.18 | Cluster Node <span class="error">&#91;akka.tcp://opendaylight-cluster-data@10.29.14.161:2550&#93;</span> - Marking node(s) as UNREACHABLE <span class="error">&#91;Member(address = akka.tcp://opendaylight-cluster-data@10.29.14.94:2550, status = Up)&#93;</span>. Node roles <span class="error">&#91;member-2&#93;</span></p> <p>Forgot the link to the failing job:</p> <p><a href="https://jenkins.opendaylight.org/releng/view/CSIT-3node/job/openflowplugin-csit-3node-periodic-bulkomatic-clustering-perf-daily-only-carbon/365/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/CSIT-3node/job/openflowplugin-csit-3node-periodic-bulkomatic-clustering-perf-daily-only-carbon/365/</a></p> <p>&gt; AskTimeoutException</p> <p>Let me guess. Ongoing transactions (read by Restconf in this case) while isolation (or JVM kill) happens?<br/> That is expected to fail while using ask-based protocol. Try using tell-based protocol (by prepending integration/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_true.robot in the testplan).</p> <p>TODO: Add a link to an official documentation on tell-based protocol here.</p> <p>This test does not take down any cluster instance, it just pushes a bunch of flows in config DS in a member that is not leader or switch master (worst case).</p> <p>Changing to tell-based protocol seems to help in carbon but not in nitrogen, not sure why. I will prepare a patch to change the test anyway.</p> <p><a href="https://git.opendaylight.org/gerrit/#/c/63592/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/63592/</a></p> <p>Luis, moving this to you for now since Sanjib is no longer active. </p> <p>I am not sure we can do much here: same perf 100K flows test works for single node but obviously performance takes a hit when cluster is used due to the backend data replication.</p> <p>I lower priority of this and I will adjust the test to pass with less flows. After that I will close the ticket.</p> <p>Test passes with 80K flows. We will leave like this for now.</p> Development External issue ID 6755 External issue URL Issue Type ODL SR Target Milestone Rank 0|i02r9j: