https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/CONTROLLER-1566 controller <p>We'd met a problem of using up JVM heap while running OpenDaylight controller. The reason was that the consuming rate of data tree change listeners was much slower than the producing rate of shard data trees. At the same time, the mailbox of actor for data tree change listener uses unlimited-size queue. So the data tree change messages had cumulated infinitely until using up the JVM heap. </p> <p>After we limited the size of the mailbox queue, the JVM won't run into the mire of fully garbage-collecting of a used-up heap and kept stable no matter how slowly the data tree change listeners were consuming their data tree change messages. </p> <p>Additionally, we added a new metric for the data tree change listener actor to track the number of dropped messages due to the limited-size queue. Attachment is the screenshot of an example of such metric.</p> <p>Operating System: All<br/> Platform: All</p> CONTROLLER-1566

The mailbox of actor for data tree change listener should use limited-size queue

Bug Resolved Won't Do Kangqian Yin Kangqian Yin Wed, 7 Dec 2016 07:12:49 +0000 Mon, 19 Dec 2016 02:33:01 +0000 Mon, 19 Dec 2016 02:33:01 +0000 0.4.0 clustering 0 3 <p>Attachment limited-length-mailbox-queue-not-handled.png has been added with description: New metric of "not-handled" for actors with mailbox type of "bounded-mailbox"</p> <p>&gt; After we limited the size of the mailbox queue</p> <p>This sounds like you already have a fix ready. If you are willing to get this fix merged:<br/> 0. Click "take" in "Assigned To" field of this Bug and set "Status" to "in progress" to let other people know this Bug has a person dedicated to fixing it.<br/> 1. Upload your patch to Gerrit and add Controller committers as reviewers.<br/> 2. Post a comment here with the Gerrit link and set "Status" to "waiting for review".<br/> 3. Work with reviewers to get you patch merged, then set status to "fixed".</p> <p>There are more verbose guides in various subpages of <a href="https://wiki.opendaylight.org/view/BestPractices" class="external-link" target="_blank" rel="nofollow noopener">https://wiki.opendaylight.org/view/BestPractices</a></p> <p>The problem with using a bounded mailbox is that change notifications will get dropped which likely isn't good either. You really should look into why the data tree change listener isn't keeping up and try to alleviate that (e.g. is it doing blocking operation(s) that could be done async?). Also maybe the front-end client(s) are producing transactions too fast.</p> <p>&gt; dropped messages due to the limited-size queue</p> <p>Oh, I missed that. No application expects data tree changes to be dropped silently.</p> <p>Would it be possible to block the committing thread (instead of dropping message) until queue is writable? Or would that lead to deadlocks (or some other queue exploding)?</p> <p>(In reply to Vratko Polák from comment #3)<br/> &gt; &gt; dropped messages due to the limited-size queue<br/> &gt; <br/> &gt; Oh, I missed that. No application expects data tree changes to be dropped<br/> &gt; silently.<br/> &gt; <br/> &gt; Would it be possible to block the committing thread (instead of dropping<br/> &gt; message) until queue is writable? Or would that lead to deadlocks (or some<br/> &gt; other queue exploding)?</p> <p>Back pressure is the ideal way to do it but in this case the committing thread would block the shard from processing subsequent messages which isn't ideal. One slow listener could block raft heartbeats and subsequent transactions.</p> <p>If a listener may do expensive processing or block I think it should offload that from the listener notification thread. If the queue or thread pool gets inundated then the listener logic can decide how best to handle that.</p> <p>Just as you said, It's unacceptable to drop data tree change messages silenty when the message queue is fulfilled. So I would add a metric to record the counter of dropped messages by overriding the class MeteredBoundedMailbox.MeteredMessageQueue. The application can monitor this metric to see whether their listners works slowly or loses some critical data tree change messages.</p> <p>And by means of this metric, we have found our bgp listeners consume data tree changes slowly than the production from the processing of bgp packets. We're now tring to offload the changes to a new queue and consolidate the changes.</p> <p>The disadvantage of limiting the queue size I've met is that it will block the shard actor for a duration of timeout when the shard tries to tell a data tree change message to a fulfilled message queue. That'll decrease throughput of odl's distributed datastore. However this is a way of back-pressure to the producers.</p> <p>This is a more complicated problem. The first it the transport part and is the inverse of BUG-5280, i.e. solution can use the same mechanism.</p> <p>As for the backpressure problem, we cannot allow the system to grind to a halt simply because a bug in some obscure application. To that effect we have introduced DOMDataTreeListener, which has two things going for it:</p> <ul class="alternate" type="square"> <li>proper error callback, i.e. "you cannot keep up, I killed you, restart if you wish"</li> <li>ability to allow notifications to be state-compressed, hence doing a similar thing PingPongDataBroker does</li> </ul> <p>These three combined can be used to:</p> <ul class="alternate" type="square"> <li>not overflow actor inbox (by exerting backpressure)</li> <li>keep the number of notifications low (state-compress them when they start piling up)</li> <li>if all else fails kill the listener</li> </ul> <p>A further note: Data</p> {Tree} <p>ChangeListeners API contract does not allow a message to be dropped, as that amounts to data corruption.</p> <p>You're right. Data corruption in application is unavoidable if the application consumes messages more slowly than the message producing from the distributed datastore. </p> <p>After we limited the listener queue length, the bgp listeners cannot work correctly for they lost many data. </p> <p>However, this will force such applications as bgp listeners to find the way to catch up with the producer. This's a good thing for it will improve the quality and throughput of whole system.</p> <p>Regarding the complexity of applications, I don't think the distributed datastore can achieve the DataTreeChangeListener contract of no message lost or no data corruption.</p> <p>The reasonable thing the distributed datastore can do on this issue is to keep more messages as possible as the JVM heap can allow rather than a fixed number of messages.</p> <p><a href="https://git.opendaylight.org/gerrit/#/c/49518/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/49518/</a></p> <p>Pushing the problem out to applications is unacceptable, simply because we do not give them the tools to solve the problem and it will lead to massive amounts of code duplication in the application layer, with the various workarounds having different kinds of bugs.</p> <p>To illustrate on the specific example of BGP listeners, how do you propose they solve the problem, when they are completely unaware of the fact that the implementation has chosen to break the DTCN stream and hence the sum of observed state no longer matches what is in the data store?</p> <p>I do not agree with your assessment of not being unable to fulfill a reasonable interface contract (where DOMDataTreeListener is reasonable). I believe we have all the tools we need to get the job done.</p> <p>Is your assessment based on the CDS architecture or its implementation deficiencies?</p> <p>Hi, Robert, What's CDS architecture? My assessment of bgp listeners comes from the performance testing of our bgp application customized from ODL's bgpcep project.</p> <p>Currently, I don't have time and ability to implement the much better solution as you said in "Comment 6".</p> <p>However, I look forward to seeing you or Tom can implement that solution to avoid using up JVM heap.</p> <p>Before we can use that solution, I have to limit the queue size because using up JVM heap is a much more dangerous thing. We've suffered it once, and we won't suffer it anymore.</p> Development External issue ID 7314 External issue URL Rank 0|i02rd3: