https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/CONTROLLER-653 controller <p>To reproduce:</p> <p>1. Put the drop test feedback to go through the MD-SAL Data Store. From the controller OSGI console, type<br/> dropAllPackets on</p> <p>2. Run cbench in the throughput mode, for example:</p> <p>cbench -c 192.168.162.1 -p 6633 -m 10000 -l 10 -s 16 -M 10000 -t</p> <p>The throughput is very low (about 2.6% of the latency cbench throughput going through the data store). After the test finishes, the CPU utilization remains very high for several minutes (I had to kill the controller after 5 min, because I ran out of time). This indicates some massive queue buildup in the system.</p> <p>I am adding thsi bug to MD-SAL because the throughput mode through RPC loopback is table, and performs very well. It's the loopback through the data store that knocks out the controller.</p> <p>Operating System: Mac OS<br/> Platform: Macintosh</p> CONTROLLER-653

Cbench throughput mode kills the MD-SAL data store

Bug Resolved Done Tom Pantelis Jan Medved Tue, 29 Jul 2014 05:42:29 +0000 Tue, 16 Sep 2014 13:39:45 +0000 Tue, 16 Sep 2014 13:39:45 +0000 Helium mdsal Thu, 7 Aug 2014 00:00:00 +0000 0 4 <p>It may be the single-threaded datastore commits that are the bottleneck. I'm working on off-loading the notifications with <a href="https://bugs.opendaylight.org/show_bug.cgi?id=1430" class="external-link" target="_blank" rel="nofollow noopener">https://bugs.opendaylight.org/show_bug.cgi?id=1430</a>.</p> <p>Unfortunately we can't get any debugging info to see the internal executor/queue stats. It would be nice to have JMX bean wrappers for the internal executors et al so we can view stats via the JConsole. I can look into that.</p> <p>I violently second the request for stats. I've asking for whatever MD-SAL stats i can get for over a year now</p> <p>I've prototyped changes to make some stats available via JMX (JConsole).</p> <p>Stats for the thread pool executors:</p> <p> activeThreadCount<br/> currentThreadPoolSize<br/> largestThreadPoolSize<br/> maxThreadPoolSize<br/> currentQueueSize<br/> maxQueueSize<br/> completedTaskCount<br/> totalTaskCount</p> <p>Stats are available for the following executors:</p> <p> Single-threaded commit coordinator<br/> Commit Future listener notification<br/> Config data store 3-phase commit<br/> Config data store DataChangeListener notification<br/> Operational data store 3-phase commit<br/> Operational data store DataChangeListener notification</p> <p>I also added commit stats:</p> <p> totalCommits<br/> longestCommitTime<br/> shortestCommitTime<br/> averageCommitTime</p> <p>Gerrits for stat work:</p> <p><a href="https://git.opendaylight.org/gerrit/#/c/9837/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/9837/</a><br/> <a href="https://git.opendaylight.org/gerrit/#/c/9797/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/9797/</a></p> <p>I added stats with this bug but it was originally created to address performance issues. There's been performance fixes since then - do we need to keep this bug open?</p> <p>There's been a lot of performance enhancements made upstream since this bug was opened and it appears the IMDS is in pretty good shape. This bug ended up being used to add stats.</p> Development External issue ID 1446 External issue URL ODL SR Target Milestone Rank 0|i02lq7: