https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/NETCONF-390 netconf <p>Single node CSIT result <span class="error">&#91;0&#93;</span> is around 400 requests per second.<br/> Three node test (device configured through node 1, device data accessed through node 3) result <span class="error">&#91;1&#93;</span> is almost 20 times worse.</p> <p><span class="error">&#91;0&#93;</span> <a href="https://logs.opendaylight.org/releng/jenkins092/netconf-csit-1node-periodic-scale-all-carbon/219/archives/log.html.gz#s1-s2-t5-k3-k15-k6-k2-k2" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/netconf-csit-1node-periodic-scale-all-carbon/219/archives/log.html.gz#s1-s2-t5-k3-k15-k6-k2-k2</a><br/> <span class="error">&#91;1&#93;</span> <a href="https://logs.opendaylight.org/sandbox/jenkins091/netconf-csit-3node-periodic-clustering-scale-all-carbon/1/archives/log.html.gz#s1-s2-t6-k3-k15-k6-k2-k2" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/sandbox/jenkins091/netconf-csit-3node-periodic-clustering-scale-all-carbon/1/archives/log.html.gz#s1-s2-t6-k3-k15-k6-k2-k2</a></p> <p>Operating System: All<br/> Platform: All</p> NETCONF-390

Netconf performance in cluster is way worse than in single node

Bug Resolved Cannot Reproduce Vratko Polak Vratko Polak Tue, 4 Apr 2017 10:30:54 +0000 Fri, 15 Mar 2019 22:22:39 +0000 Fri, 21 Apr 2017 08:36:40 +0000 netconf 0 4 <p>Could you pleas rerun sandbox job for 3node clustering</p> <p>The performance for Carbon <del>all</del> test is now known <span class="error">&#91;0&#93;</span> to be around 20.48 requests per second. Interestingly, Carbon <del>only</del> test has slightly worse performance of around 19.26 requests per second, result from Sandbox <span class="error">&#91;1&#93;</span>.</p> <p><span class="error">&#91;0&#93;</span> <a href="https://logs.opendaylight.org/releng/jenkins092/netconf-csit-3node-periodic-clustering-scale-all-carbon/228/archives/log.html.gz#s1-s2-t6-k3-k15-k6-k2-k2" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/netconf-csit-3node-periodic-clustering-scale-all-carbon/228/archives/log.html.gz#s1-s2-t6-k3-k15-k6-k2-k2</a><br/> <span class="error">&#91;1&#93;</span> <a href="https://logs.opendaylight.org/sandbox/jenkins091/netconf-csit-3node-periodic-clustering-scale-only-carbon/1/archives/log.html.gz#s1-s2-t6-k3-k15-k6-k2-k2" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/sandbox/jenkins091/netconf-csit-3node-periodic-clustering-scale-only-carbon/1/archives/log.html.gz#s1-s2-t6-k3-k15-k6-k2-k2</a></p> <p>Is this a comparison between the clustered and non-clustered feature or clustered feature running in single and 3 node scenarios?</p> <p>Could you also send me what parameters are you using for testtool and RestPerfClient, please?</p> <p>I am not really sure how you could achieve such a big gap in those tests because when I am testing it localy I get only 2 to 5 requests difference. My performance is 17 to 20 requests for netconf-topology and 15 to 17 requests for netconf-clustered-topology.</p> <p>Only time I was able to receive higher amount of requests, was when I was sending requests to mountpoint which was not created and therefore could not write anything. I was just wondering if this can`t be also case of the robot test. Or something similiar.</p> <p>&gt; Is this a comparison between the clustered and non-clustered feature</p> <p>Looking at the job definition: Yes, it is.<br/> Should the clustered feature be that much slower?</p> <p>&gt; what parameters are you using for testtool and RestPerfClient</p> <p>Beryllium commands (identical for single node and cluster, except log file name):<br/> Testtool start <span class="error">&#91;4&#93;</span> (nothing in ./schemas):<br/> /usr/lib/jvm/java-7-openjdk-amd64/bin/java <del>Xmx1G -XX:MaxPermSize=256M -Dorg.apache.sshd.registerBouncyCastle=false -jar netconf-testtool-1.0.5-20170413.092814-27-executable.jar --device-count 1 --debug false --schemas-dir ./schemas --md-sal false &gt;testtool</del>-netconf-scale-txt-Performance.log 2&gt;&amp;1</p> <p>Restperfclient start <span class="error">&#91;5&#93;</span>:<br/> /usr/lib/jvm/java-7-openjdk-amd64/bin/java <del>Xmx1G -XX:MaxPermSize=256M -jar netconf-testtool-1.0.5-20170413.092814-27-rest-perf-client.jar --ip 10.29.13.188 --port 8181 --edits 65536 --edit-content request1.json --async-requests true --auth admin admin --timeout 22 --destination /restconf/config/network-topology:network-topology/topology/topology-netconf/node/17830-sim-device/yang-ext:mount/car:cars 2&gt;&amp;1 | tee restperfclient</del>-netconf-scale-txt-Performance.log</p> <p>request1.json is here <span class="error">&#91;6&#93;</span>.</p> <p><span class="error">&#91;4&#93;</span> <a href="https://logs.opendaylight.org/releng/jenkins092/netconf-csit-1node-periodic-scale-only-beryllium/456/archives/log.html.gz#s1-s2-t1-k3-k7" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/netconf-csit-1node-periodic-scale-only-beryllium/456/archives/log.html.gz#s1-s2-t1-k3-k7</a><br/> <span class="error">&#91;5&#93;</span> <a href="https://logs.opendaylight.org/releng/jenkins092/netconf-csit-1node-periodic-scale-only-beryllium/456/archives/log.html.gz#s1-s2-t5-k3-k15-k4" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/netconf-csit-1node-periodic-scale-only-beryllium/456/archives/log.html.gz#s1-s2-t5-k3-k15-k4</a><br/> <span class="error">&#91;6&#93;</span> <a href="https://github.com/opendaylight/integration-test/blob/master/csit/variables/netconf/RestPerfClient/request1.json" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/opendaylight/integration-test/blob/master/csit/variables/netconf/RestPerfClient/request1.json</a></p> <p>(In reply to Vratko Polák from comment #5)<br/> &gt; &gt; Is this a comparison between the clustered and non-clustered feature<br/> &gt; <br/> &gt; Looking at the job definition: Yes, it is.<br/> &gt; Should the clustered feature be that much slower?<br/> &gt; </p> <p>Do we know if the requests are going through the slave mountpoint? The master mountpoint should have similar performance numbers as the nonclustered feature<br/> as there's no overhead afaik, I can only see this happening on the slave mountpoints and I'm not sure there's space for performance improvements there since it's only forwarding the requests.</p> <p>Could you try to start Restperfclient with asynRequests --async-requests false??? for both single node and 3 node cluster?</p> <p><a href="https://jenkins.opendaylight.org/releng/view/netconf/job/netconf-csit-1node-periodic-scale-only-carbon/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/netconf/job/netconf-csit-1node-periodic-scale-only-carbon/</a><br/> 1node runs Rest perf client with --async-requests true</p> <p><a href="https://jenkins.opendaylight.org/releng/view/netconf/job/netconf-csit-3node-periodic-clustering-scale-only-carbon/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/netconf/job/netconf-csit-3node-periodic-clustering-scale-only-carbon/</a><br/> 3node runs Rest perf client with --async-requests false</p> <p>That's the reason of poor performance. In current implementation of netconf clustered topology it isn't possible to use async requests, since it permits only one open write transaction at the time and async requests would cause IllegalStateExceptions - Transaction is already opened for another user.</p> <p><a href="https://git.opendaylight.org/gerrit/#/c/52235/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/52235/</a> would allow to use async requests in tests. However, simultaneous write on real device via either odl-netconf-topology or odl-netconf-clustered-topology would fail, because lock rpc is sent to device before every write and unlock is sent when write completes.</p> <p>&gt; Do we know if the requests are going through the slave mountpoint?</p> <p>Requests are doing through member-3, which is practically always a slave.</p> <p><a href="https://jenkins.opendaylight.org/releng/view/netconf/job/netconf-csit-1node-periodic-scale-only-carbon/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/netconf/job/netconf-csit-1node-periodic-scale-only-carbon/</a> should be adapted to use --async-requests false</p> <p>&gt; use --async-requests false</p> <p>Suite change <span class="error">&#91;0&#93;</span> tested on Sandbox <span class="error">&#91;1&#93;</span> confirms the (sync) performance is basically the same on 1 and 3 nodes.</p> <p><span class="error">&#91;0&#93;</span> <a href="https://git.opendaylight.org/gerrit/#/c/55707/2" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/55707/2</a><br/> <span class="error">&#91;1&#93;</span> <a href="https://jenkins.opendaylight.org/sandbox/job/netconf-csit-1node-periodic-scale-all-carbon/4/robot/netconf-scale.txt/Performance/Run_Restperfclient/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/sandbox/job/netconf-csit-1node-periodic-scale-all-carbon/4/robot/netconf-scale.txt/Performance/Run_Restperfclient/</a></p> Development External issue ID 8149 External issue URL Issue Type Rank 0|i01y8v: