https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/GENIUS-96 genius <p>Internal downstream testing reports OOM with latest stable/nitrogen builds.</p> <p>HPROF analysis by MAT points to something really badly wrong in DataStoreJobCoordinator.</p> <p>see attached ZIP</p> GENIUS-96

DataStoreJobCoordinator OOM

Bug Highest Verified Cannot Reproduce Michael Vorburger Michael Vorburger Fri, 3 Nov 2017 21:51:32 +0000 Thu, 19 Apr 2018 14:41:46 +0000 Thu, 19 Apr 2018 14:41:46 +0000 Nitrogen-SR1 Nitrogen-SR1 General 0 5 <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=tpantelis" class="user-hover" rel="tpantelis">tpantelis</a> dixit:</p> <blockquote><p>maybe some job got stuck or took a long time and backed up a queue</p></blockquote> <p>This somehow sounds familiar - didn't you mention something like this somewhere recently, <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=k.faseela" class="user-hover" rel="k.faseela">k.faseela</a> ?</p> <p>BTW this reminds me that during the code reviews of the genius DJC to infrautils JC we added this to the JavaDoc:</p> <blockquote><p>Enqueued jobs are stored in unbounded queues until they are run, this should be kept in mind as it might lead to an OOM.</p></blockquote> <p>and now that ^^^ is exactly what happened here... it would probably be better if we made the JobCoordinator reject jobs after a certain (configurable) capacity? And start dumping whatever is stuck in the queue at that point?</p> <p>Yeah..this is what even I was proposing that day. <br/> In one of our testing, saw that two Jobs were in deadlock, and enough of such jobs were queued up to take up all available threads, and none of the new jobs had resources to get executed. <br/> Taking a jstack output at that time helped us understand the culprit.</p> <p><a href="https://jira.opendaylight.org/browse/INFRAUTILS-24" title="JobCoordinator should use bounded Executors and bound its own job queue" class="issue-link" data-issue-key="INFRAUTILS-24"><del>INFRAUTILS-24</del></a> will make the JobCoordinator reject jobs to avoid the such OOMs in the future. We need to fix the bug hit during testing in this JIRA issue; I'll get jstack.</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=k.faseela" class="user-hover" rel="k.faseela">k.faseela</a> said on IRC that earlier patch sets of <a href="https://git.opendaylight.org/gerrit/#/c/63884/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/63884/</a> caused this kind of problem, but that the final one (which also went into nitrogen) "should" have fixed the deadlock. So when we get the jstack, we should watch out whether it looks like it could have anything to do with that change, just to be sure. (She also mentioned that there will be a follow up patch which will fix something re. a "slowing down of the DJC", but that's not related to deadlock / OOM, AFAIK.)</p> <p><a href="https://git.opendaylight.org/gerrit/65146" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/65146</a></p> <p>This will have some impact?</p> <p>Kency indicated that there were some lockmanager related issues which used to make some jobs stuck in DJC, which is fixed under below review :</p> <p><a href="https://git.opendaylight.org/gerrit/#/c/61977/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/61977/</a></p> <p>Could you please review and merge?</p> <p>Is this issue related to <a href="https://jira.opendaylight.org/browse/NETVIRT-974" class="external-link" rel="nofollow">https://jira.opendaylight.org/browse/NETVIRT-974</a> ?</p> <p>Is this truly a blocker for Nitrogen-SR1? Thanks!</p> <p>&gt; Is this issue related to <a href="https://jira.opendaylight.org/browse/NETVIRT-974" class="external-link" rel="nofollow">https://jira.opendaylight.org/browse/NETVIRT-974</a> ?</p> <p>no, not at all. But it could turn out to have one and the same single cause as <a href="https://jira.opendaylight.org/browse/GENIUS-97" title="IdPoolListener OOM" class="issue-link" data-issue-key="GENIUS-97"><del>GENIUS-97</del></a> though - we don't know yet.</p> <p>&gt; Is this truly a blocker for Nitrogen-SR1? Thanks!</p> <p>Yup.</p> <p>Stack trace just attached to <a href="https://jira.opendaylight.org/browse/GENIUS-97" title="IdPoolListener OOM" class="issue-link" data-issue-key="GENIUS-97"><del>GENIUS-97</del></a> shows a lot of DataStoreJobCoordinator ...</p> <p>This can only be reproduced with latest stable/Nitrogen HEAD (which will be SR1), NOT with the first September 26 Nitro, so recently broke.</p> <p>Do we have ETA on resolution? Need input to assess how far we have to push out Nitrogen-SR1. Thanks!</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=klou" class="user-hover" rel="klou">klou</a> ETA is when it's Fixed. We can try to do earlier then when it's Done, but we would need a time machine.</p> <p>Closing as CANNOT REPRO, because <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=ltomasbo" class="user-hover" rel="ltomasbo">ltomasbo</a> has clarified that he only hits an OOM (reproducible) on Nitrogen 0.7.0 and 0.7.1 (=SP1) with Xmx 512 MB heap, instead of the default 2 GB, using which ie works for him. The 512 MB is the default when deploying ODL with devstack without specifying a different Xmx. We're proposing to fix that in <a href="https://review.openstack.org/#/c/518540/" class="external-link" target="_blank" rel="nofollow noopener">https://review.openstack.org/#/c/518540/</a> to avoid future confusion.</p> <p>PS: We'll be adding proper JobCoordinator monitor-ability via <a href="https://jira.opendaylight.org/browse/INFRAUTILS-19" title="Coda Hale Dropwizard Metrics integration" class="issue-link" data-issue-key="INFRAUTILS-19"><del>INFRAUTILS-19</del></a>.</p> Blocks INFRAUTILS-24 Relates INFRAUTILS-19 Development Priority Rank 0|i0378v: