https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/NETCONF-170 netconf <p>When RestPerfClient is run on MDSAL Northbound directly, the MDSAL Northbound manages to serve over 400 requests per second. However when it is run on MDSAL Northbound mapped on a Netconf Connector, the request rate drops significantly. The RestPerfClient output log attached to <a href="https://jira.opendaylight.org/browse/NETCONF-169" title="RestPerfClient: Invalid requests rate reported when a thread times out" class="issue-link" data-issue-key="NETCONF-169"><del>NETCONF-169</del></a> suggests it is below 220 per second but the exact rate could not be determined because of <a href="https://jira.opendaylight.org/browse/NETCONF-169" title="RestPerfClient: Invalid requests rate reported when a thread times out" class="issue-link" data-issue-key="NETCONF-169"><del>NETCONF-169</del></a> combined with lack of support for configuring the timeouts in RestPerfClient (which is a new feature) in the performance test suite.</p> <p>Yes, I expect a performance drop when going through a Netconf connector versus going directly but a performance drop of over 50% is way too severe.</p> <p>Looking into the logs reveals massive amount of "Timing out for 1 session(s) with id &lt;24-to-26-random-chars&gt;" warnings emitted during the time when RestPerfClient emitting the requests to the Netconf connector mapped MDSAL Northbound.</p> <p>Operating System: All<br/> Platform: All</p> NETCONF-170

Poor performance of MDSAL Northbound mapped on a Netconf connector

Bug Resolved Done Unassigned Jozef Behran Tue, 22 Mar 2016 09:34:46 +0000 Fri, 15 Mar 2019 22:22:22 +0000 Thu, 13 Apr 2017 11:48:55 +0000 netconf 0 6 <p>Hi, if you compare steps executed during direct and mountpoint access to ODL datastore, you can see, that access via netconf connector is much more complex.</p> <p>Direct approach:<br/> 1. HTTP server<br/> 2. URL, XXL -&gt; YangInstanceIdentifier, NormalizedNode<br/> 3. Datastore write</p> <p>Via netconf connector<br/> 1. HTTP server<br/> 2. URL, XXL -&gt; YangInstanceIdentifier, NormalizedNode<br/> 3. YangInstanceIdentifier, NormalizedNode -&gt; Netconf RPC XML<br/> 4. Netconf southbound Netty client - exi handlers, ssh, network io<br/> 5. Netconf northbound Netty server - exi handlers, ssh, network io<br/> 6. Netconf RPC XML -&gt; YangInstanceIdentifier, NormalizedNode<br/> 7. Datastore write</p> <p>Additionally, netconf mountpoint write transaction operations are implemented as sequence of multiple netconf rpcs (lock, edit-config, commit, unlock)<br/> which causes additional network io overhead.</p> <p>"Timing out for 1 session(s) with id &lt;24-to-26-random-chars&gt;" warnings are logged by restconf and shouldn't affect this issue.</p> <p>I've also profiled ODL under this test and haven't found spots suitable for optimalization.</p> <p>Attachment 5581.zip has been added with description: job files</p> <p>All the scale jobs are hitting the same problem, that sending rest requests using netconf connector takes too long.<br/> At the moment (Jan03) it takes so long that ssh session is broken during the test and we dont see the poor performance.<br/> All jobs affected: netconf-csit-1node-periodic-scale-&lt;all|only&gt;-&lt;beryllium,boron,carbon&gt;</p> <p>Using refs/changes/68/49468/8 the test was done and all files are attached. The performance drop is seen comparing restperfclient-<del>netconf-scale-txt-Mdsal</del><del>direct.log.gz and restperfclient</del><del>netconf-scale-txt-Mdsal</del>-netconf-connector.log.gz.</p> <p>13:11:03.692 <span class="error">&#91;main&#93;</span> INFO o.o.n.t.t.c.http.perf.RestPerfClient - FINISHED. Execution time: 11.66 min<br/> 13:11:03.698 <span class="error">&#91;main&#93;</span> INFO o.o.n.t.t.c.http.perf.RestPerfClient - Requests per second: 7.148453360630894<br/> vs.<br/> 2:59:21.266 <span class="error">&#91;main&#93;</span> INFO o.o.n.t.t.c.http.perf.RestPerfClient - FINISHED. Execution time: 20.38 s<br/> 12:59:21.274 <span class="error">&#91;main&#93;</span> INFO o.o.n.t.t.c.http.perf.RestPerfClient - Requests per second: 245.30245793062846</p> <p>Drop in performance from 245 req/s to 7 req/s is too significant even if the netconf connector path is more complicated.</p> <p>Exi handler optimization<br/> <a href="https://git.opendaylight.org/gerrit/#/c/51834/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/51834/</a><br/> Merge empty list instead of exist-check<br/> <a href="https://git.opendaylight.org/gerrit/#/c/51866/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/51866/</a><br/> If it is possible, read only list entry instead of whole list for filtering<br/> <a href="https://git.opendaylight.org/gerrit/#/c/51643/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/51643/</a></p> <p>With these patches, test shouldn't fail on timeout and it should make ~22 requests/s. <br/> When ssh is disabled, test can do more than 100 reqests/s, so there may be spots for further optimization in netconf SSH handling.</p> <p>I think the results are fine. We know the SSH integration needs major surgery to get the performance we would like, especially in case of single-threaded access.</p> <p>Netconf SSH pipeline improvements are tracked here:<br/> <a href="https://bugs.opendaylight.org/show_bug.cgi?id=2484" class="external-link" target="_blank" rel="nofollow noopener">https://bugs.opendaylight.org/show_bug.cgi?id=2484</a></p> Development External issue ID 5581 External issue URL Issue Type Rank 0|i01wvz: