https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/NETCONF-355 netconf <p>The regression may be in clustering code, or in Restconf code.<br/> This affects both Boron and Carbon snapshots, in comparison with Beryllium snapshots.</p> <p>The biggest difference is in Add_Cars test case, which uses a script <span class="error">&#91;0&#93;</span> to send single POST request to create 10000 car items. But also Add_People test case takes longer, compare Boron durations <span class="error">&#91;1&#93;</span> to Beryllium <span class="error">&#91;2&#93;</span>.</p> <p><span class="error">&#91;0&#93;</span> <a href="https://github.com/opendaylight/integration-test/blob/master/tools/odl-mdsal-clustering-tests/scripts/cluster_rest_script.py" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/opendaylight/integration-test/blob/master/tools/odl-mdsal-clustering-tests/scripts/cluster_rest_script.py</a><br/> <span class="error">&#91;1&#93;</span> <a href="https://jenkins.opendaylight.org/releng/view/controller/job/controller-csit-1node-rest-cars-perf-only-boron/870/robot/controller-rest-cars-perf.txt/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/controller/job/controller-csit-1node-rest-cars-perf-only-boron/870/robot/controller-rest-cars-perf.txt/</a><br/> <span class="error">&#91;2&#93;</span> <a href="https://jenkins.opendaylight.org/releng/view/controller/job/controller-csit-1node-rest-cars-perf-only-beryllium/116/robot/controller-rest-cars-perf.txt/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/controller/job/controller-csit-1node-rest-cars-perf-only-beryllium/116/robot/controller-rest-cars-perf.txt/</a></p> <p>Operating System: All<br/> Platform: All</p> NETCONF-355

Performance regression in controller-rest-cars-perf

Bug Verified Done Robert Varga Vratko Polak Tue, 28 Feb 2017 14:22:16 +0000 Fri, 15 Mar 2019 22:22:36 +0000 Fri, 21 Apr 2017 11:02:29 +0000 restconf-nb 0 4 <p>Well, Verify_Cars fails, can they be related?</p> <p>To quantify this a bit, we are seeing about 100% performance degradation observed from test duration.</p> <p>&gt; Verify_Cars fails</p> <p>I do not see that in recent builds. Links without specific build number:<br/> Be <span class="error">&#91;3&#93;</span>, B <span class="error">&#91;4&#93;</span>, C <span class="error">&#91;5&#93;</span>.</p> <p>Note that those are single node tests, the suite runs also in three node setup <span class="error">&#91;6&#93;</span>, but the difference in performance is small.</p> <p><span class="error">&#91;3&#93;</span> <a href="https://jenkins.opendaylight.org/releng/view/controller/job/controller-csit-1node-rest-cars-perf-only-beryllium/lastSuccessfulBuild/robot/controller-rest-cars-perf.txt/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/controller/job/controller-csit-1node-rest-cars-perf-only-beryllium/lastSuccessfulBuild/robot/controller-rest-cars-perf.txt/</a><br/> <span class="error">&#91;4&#93;</span> <a href="https://jenkins.opendaylight.org/releng/view/controller/job/controller-csit-1node-rest-cars-perf-only-boron/lastSuccessfulBuild/robot/controller-rest-cars-perf.txt/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/controller/job/controller-csit-1node-rest-cars-perf-only-boron/lastSuccessfulBuild/robot/controller-rest-cars-perf.txt/</a><br/> <span class="error">&#91;5&#93;</span> <a href="https://jenkins.opendaylight.org/releng/view/controller/job/controller-csit-1node-rest-cars-perf-only-carbon/lastSuccessfulBuild/robot/controller-rest-cars-perf.txt/010%20Crud%20Mdsal%20Perf/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/controller/job/controller-csit-1node-rest-cars-perf-only-carbon/lastSuccessfulBuild/robot/controller-rest-cars-perf.txt/010%20Crud%20Mdsal%20Perf/</a><br/> <span class="error">&#91;6&#93;</span> <a href="https://jenkins.opendaylight.org/releng/view/controller/job/controller-csit-3node-rest-clust-cars-perf-only-carbon/lastSuccessfulBuild/robot/controller-rest-clust-cars-perf.txt/010%20Crud%20Mdsal%20Perf/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/controller/job/controller-csit-3node-rest-clust-cars-perf-only-carbon/lastSuccessfulBuild/robot/controller-rest-clust-cars-perf.txt/010%20Crud%20Mdsal%20Perf/</a></p> <p>Numeric values (for times when Beryllium jobs are removed) for Add_Cars:<br/> Beryllium: 0:00:28.534<br/> Boron: 0:01:24.138<br/> Carbon: 0:01:36.001</p> <p>Testing with distribution-karaf-0.6.0-20170418.121829-4936, it looks like the add case is doing an awful lot of reads.</p> <p>I will profile this further to get to the bottom of this.</p> <p>Add must check for existence (by read) but reads were also in beryllium,<br/> by comparison of code there are exactly the same operations on transactions (merge, read, put) in master as were done in beryllium.</p> <p>There are some modifications in master compare to beryllium (uri parameters) but in this simple case they are not applied.</p> <p>When profiled beryllium and master and compared there were a lot of changes in Yangtools, maybe added some additional verifications for data.</p> <p>Did you find more reads in recent versions?</p> <p>Well, the reason for the regression is in RESTCONF. The regression was introduced in <a href="https://git.opendaylight.org/gerrit/#/c/30554/48/restconf/sal-rest-connector/src/main/java/org/opendaylight/netconf/sal/restconf/impl/BrokerFacade.java@320" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/30554/48/restconf/sal-rest-connector/src/main/java/org/opendaylight/netconf/sal/restconf/impl/BrokerFacade.java@320</a>, as now BrokerFacade issues a read request for each individual item in the list.</p> <p>So, the question is: what exactly is the reason for issuing the massive amounts of reads? I would prefer exact quotes from drafts, because the code sure as hell does not contain any sort of documentation about what is going on.</p> <p>Attachment Main-2017-04-18-2.snapshot.xz has been added with description: YKP tracing snapshot</p> <p>Snapshot shows BrokerFacade executing 10K read operations. Each of those is being waited synchronously, which is not something you can do while expecting reasonable performance.</p> <p>Furthermore, these reads are executed as the target is being modified, which is forcing DataTree to apply operations so it can report accurate results.</p> <p>easy data - POST empty container<br/> complete data - POST list items into container</p> <p>Attachment time results.tar.gz has been added with description: result of time measure between Be-4 and master running the same data</p> <p>(In reply to Robert Varga from comment #7)<br/> &gt; Well, the reason for the regression is in RESTCONF. The regression was<br/> &gt; introduced in<br/> &gt; <a href="https://git.opendaylight.org/gerrit/#/c/30554/48/restconf/sal-rest-connector/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/30554/48/restconf/sal-rest-connector/</a><br/> &gt; src/main/java/org/opendaylight/netconf/sal/restconf/impl/BrokerFacade.<br/> &gt; java@320, as now BrokerFacade issues a read request for each individual item<br/> &gt; in the list.</p> <p>-this code is for PACTH operation, we are dealing now with POST (postDataViaTransaction method)</p> <p>when batch size for command is used wall time difference between master and be-4 arises</p> <p>command:<br/> /usr/bin/time -v python cluster_rest_script.py --host localhost --port 8181 --threads 1 --itemcount 10000 --ipr 10000 --debug add</p> <p>Attachment compare has been added with description: compare be-4 and master when batch size is 10000</p> <p>carbon: <a href="https://git.opendaylight.org/gerrit/55205" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/55205</a><br/> boron: <a href="https://git.opendaylight.org/gerrit/55211" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/55211</a></p> <p>This will need a proper follow-up to address overall code structure like:</p> <ul class="alternate" type="square"> <li>eliminate blocking checks in loops, so the checks are done in parallel</li> <li>make RestDocumentedException a checked exception</li> <li>audit all code to make sure checks are done before modifications</li> <li>ask controller/mdsal to add a create() method</li> </ul> <p>will sit down with Ivan &amp; Jakub to discuss follow-up notes</p> <p>Boron looks fixed <span class="error">&#91;7&#93;</span>:<br/> Duration: 0:00:02.544 (-0:01:18.522)</p> <p>Carbon merged, waiting for test run.<br/> Nitrogen <span class="error">&#91;8&#93;</span> not merged just yet.</p> <p><span class="error">&#91;7&#93;</span> <a href="https://jenkins.opendaylight.org/releng/view/controller/job/controller-csit-1node-rest-cars-perf-only-boron/947/robot/controller-rest-cars-perf.txt/Add%20Cars/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/controller/job/controller-csit-1node-rest-cars-perf-only-boron/947/robot/controller-rest-cars-perf.txt/Add%20Cars/</a><br/> <span class="error">&#91;8&#93;</span> <a href="https://git.opendaylight.org/gerrit/55231" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/55231</a></p> <p>&gt; Nitrogen <span class="error">&#91;8&#93;</span></p> <p>merged now.</p> <p>New Carbon performance <span class="error">&#91;9&#93;</span> number: 0:00:02.314</p> <p><span class="error">&#91;9&#93;</span> <a href="https://jenkins.opendaylight.org/releng/view/controller/job/controller-csit-1node-rest-cars-perf-all-carbon/245/robot/controller-rest-cars-perf.txt/010%20Crud%20Mdsal%20Perf/Add%20Cars/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/controller/job/controller-csit-1node-rest-cars-perf-all-carbon/245/robot/controller-rest-cars-perf.txt/010%20Crud%20Mdsal%20Perf/Add%20Cars/</a></p> <p>Follow up bugs/improvements:</p> <p><a href="https://bugs.opendaylight.org/show_bug.cgi?id=8268" class="external-link" target="_blank" rel="nofollow noopener">https://bugs.opendaylight.org/show_bug.cgi?id=8268</a><br/> <a href="https://bugs.opendaylight.org/show_bug.cgi?id=8269" class="external-link" target="_blank" rel="nofollow noopener">https://bugs.opendaylight.org/show_bug.cgi?id=8269</a><br/> <a href="https://bugs.opendaylight.org/show_bug.cgi?id=8270" class="external-link" target="_blank" rel="nofollow noopener">https://bugs.opendaylight.org/show_bug.cgi?id=8270</a><br/> <a href="https://bugs.opendaylight.org/show_bug.cgi?id=8271" class="external-link" target="_blank" rel="nofollow noopener">https://bugs.opendaylight.org/show_bug.cgi?id=8271</a><br/> <a href="https://bugs.opendaylight.org/show_bug.cgi?id=8272" class="external-link" target="_blank" rel="nofollow noopener">https://bugs.opendaylight.org/show_bug.cgi?id=8272</a></p> <p>Also performance issue: <a href="https://bugs.opendaylight.org/show_bug.cgi?id=6412" class="external-link" target="_blank" rel="nofollow noopener">https://bugs.opendaylight.org/show_bug.cgi?id=6412</a></p> Development External issue ID 7868 External issue URL Issue Type ODL SR Target Milestone Rank 0|i01y13: