https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/CONTROLLER-263 controller <p>MDSAL Default Notification pool is currently configured as follows:<br/> Core threads: 4<br/> Max threads : 32<br/> Queue: Unbounded Linkedblockingqueue</p> <p>During cbench tests, its observed the # of threads working on the queue, dont ever go beyond 4. <br/> It should ideally hit 32, as the notification Q is not getting drained at a faster rate.</p> <p>Operating System: Mac OS<br/> Platform: Macintosh</p> CONTROLLER-263

MDSAL Notification pool threads not getting spawned besides the core threads

Bug Resolved Done Kamal Rameshan Kamal Rameshan Tue, 1 Apr 2014 22:47:48 +0000 Tue, 25 Jul 2023 08:23:22 +0000 Wed, 23 Apr 2014 21:47:15 +0000 mdsal 0 3 <p>The threadpoolexecutor does not create new threads if the queue is not full.</p> <p>Since the notification queue is unbounded, the above condition is never met and hence no new threads are spawned (besides the 4 core ones)</p> <p>I am not sure whether having a unbounded queue is the correct behavior here.</p> <p>Sorry for incomplete comment.</p> <p>While I see the problem here, and the fix makes for some better throughput, I am afraid we are not introducing a backpressure mechanism and at some point we'll see producers overwhelm consumers, creating a huge backlog.</p> <p>I think we need a more full solution, probably create a per-producer queue and some sort of fair queue dispatch threadpool, which would:</p> <p>1) prevent a single producer from monopolizing the notification dispatch threadpool<br/> 2) apply backpressure to heavy producers, such that we don't end up with huge backlogs</p> <p>I am thinking something along the lines of token buckets.</p> <p>Tony: where would be a good place to place such a rate-limiting mechanism?</p> <p>Kamal: can you introduce a knob, which would place a limit on the queue? ThreadPoolExecutor.CallerRunsPolicy looks like a good first cut at the rejection policy in that case.</p> <p>The patch is to fix a bug in the current design. </p> <p>I am in total agreement that an unbounded queue is not the approach we want to take as an OOM error is not far off.</p> <p>Rather than creating a queue/producer, my take would be to put a RateLimiter(com.google.common.util.concurrent) for each producer. <br/> And the queue could be a bounded, possibly with a higher number.</p> <p>What would be the throttling logic is something we need to come up with.</p> <p>I can take up this as an enhancement, if we agree on the design.</p> <p>But for now, i think we should have a fix for the threadpool. Currently its broken.</p> <p>Working on a patch to implement CallerRunsPolicy or any other way to throttle.</p> <p>Will run some cbench tests to see if it impacts significantly.</p> <p>CallerRunsPolicy implementation is causing thread boundaries to be violated and also having performance issues.</p> <p>Will raise a separate bug as 'enhancement' to introduce a throttle as Robert suggested.</p> <p>For now just fixing the patch for the service release: </p> <p><a href="https://git.opendaylight.org/gerrit/#/c/5857/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/5857/</a></p> Development External issue ID 645 External issue URL Priority Rank 0|i02jbj: