https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/NETVIRT-937 netvirt <p><a href="https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-ocata-gate-stateful-carbon/105/odl1_karaf.log.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-ocata-gate-stateful-carbon/105/odl1_karaf.log.gz</a></p> <p>2017-10-03 20:15:53,092 | INFO | nPool-1-worker-2 | ElanInstanceManager | 341 - org.opendaylight.netvirt.elanmanager-impl - 0.4.2.SNAPSHOT | Deleting the elanInterface present under ConfigDS:97fe230c-b83b-4bb4-bfc8-af085a471e92<br/> 2017-10-03 20:15:53,092 | INFO | nPool-1-worker-2 | ElanInstanceManager | 341 - org.opendaylight.netvirt.elanmanager-impl - 0.4.2.SNAPSHOT | unbind the Interface:97fe230c-b83b-4bb4-bfc8-af085a471e92 service bounded to Elan:8f9ddf92-e04d-4e60-81c0-6bc8860d788b<br/> 2017-10-03 20:15:53,092 | ERROR | nPool-1-worker-1 | DataStoreJobCoordinator | 292 - org.opendaylight.genius.mdsalutil-api - 0.2.2.SNAPSHOT | Thread terminated due to uncaught exception: ForkJoinPool-1-worker-1<br/> java.lang.NullPointerException: at index 0<br/> at com.google.common.collect.ObjectArrays.checkElementNotNull(ObjectArrays.java:240)<span class="error">&#91;38:com.google.guava:18.0.0&#93;</span><br/> at com.google.common.collect.ObjectArrays.checkElementsNotNull(ObjectArrays.java:231)<span class="error">&#91;38:com.google.guava:18.0.0&#93;</span><br/> at com.google.common.collect.ObjectArrays.checkElementsNotNull(ObjectArrays.java:226)<span class="error">&#91;38:com.google.guava:18.0.0&#93;</span><br/> at com.google.common.collect.ImmutableList.construct(ImmutableList.java:303)<span class="error">&#91;38:com.google.guava:18.0.0&#93;</span><br/> at com.google.common.collect.ImmutableList.copyOf(ImmutableList.java:258)<span class="error">&#91;38:com.google.guava:18.0.0&#93;</span><br/> at com.google.common.collect.ImmutableList.copyOf(ImmutableList.java:226)<span class="error">&#91;38:com.google.guava:18.0.0&#93;</span><br/> at com.google.common.util.concurrent.Futures.allAsList(Futures.java:1027)<span class="error">&#91;38:com.google.guava:18.0.0&#93;</span><br/> at org.opendaylight.genius.datastoreutils.DataStoreJobCoordinator$MainTask.run(DataStoreJobCoordinator.java:297)<span class="error">&#91;292:org.opendaylight.genius.mdsalutil-api:0.2.2.SNAPSHOT&#93;</span><br/> at java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1402)<span class="error">&#91;:1.8.0_141&#93;</span><br/> at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)<span class="error">&#91;:1.8.0_141&#93;</span><br/> at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)<span class="error">&#91;:1.8.0_141&#93;</span><br/> at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)<span class="error">&#91;:1.8.0_141&#93;</span><br/> at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)<span class="error">&#91;:1.8.0_141&#93;</span></p> <p>Operating System: All<br/> Platform: All</p> NETVIRT-937

DataStoreJobCoordinator java.lang.NullPointerException: at index 0

Bug Resolved Done Michael Vorburger Sam Hague Wed, 4 Oct 2017 00:17:42 +0000 Wed, 12 Dec 2018 12:27:57 +0000 Tue, 28 Nov 2017 17:46:31 +0000 Carbon General 0 6 <p>Attachment odl1_karaf.log.tar.xz has been added with description: odl1_karaf.log</p> <p>older job with issue: <a href="https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-ocata-gate-stateful-oxygen/89/odl1_karaf.log.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-ocata-gate-stateful-oxygen/89/odl1_karaf.log.gz</a></p> <p>Let me see if I can figure out what is going on in this one here...</p> <p>+Cc Faseela, Vivek &amp; David, who were on private email thread re. this:</p> <p><a href="https://git.opendaylight.org/gerrit/#/c/63965/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/63965/</a> illustrates via new @Test bug9238CallableListWithNull() how this can happen in JobCoordinator.. that test fails with the exact same stack trace as the one Sam provided above, from a CSIT. Basically, we see that some code somewhere is returning a (List containining a) null ListenableFuture. I suspect its a bug where someone wanted to return a completed ListenableFuture of value null, but instead directly returned a null... <img class="emoticon" src="https://jira.opendaylight.org/images/icons/emoticons/sad.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>The trouble is that we cannot yet know what the culprit for this is... but c/63965 also adds additional logging, which should help us find out. </p> <p><a href="https://git.opendaylight.org/gerrit/#/c/63966/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/63966/</a> then does the same in genius' DataStoreJobCoordinator as c/63965 does in infrautils' JobCoordinatorImpl. (And having had to do this twice is further motivation for finally wrapping up <a href="https://git.opendaylight.org/gerrit/#/q/topic:genius.DJC-to-infra.JC" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/q/topic:genius.DJC-to-infra.JC</a>!)</p> <p>Once c/63965 and c/63966 will have been merged, you'll get another exception in logs (""Runnnable likely about to terminate thread due to uncaught exception; but here is useful debugging context:"), please post that here when seen and that will then tell us where the root cause for this is where a null List element is returned by mistake, which should then let us fix this for good (easily).</p> <p>BTW: <a href="https://git.opendaylight.org/gerrit/#/c/63961/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/63961/</a> has a more medium term solution which will help detect the root cause of this problem in the future at code/build instead of runtime!! But there's a lot more work in all of genius &amp; netvirt TBD until we'll reach that Nirvana.. but it's still a step.</p> <p>PS: <a href="https://git.opendaylight.org/gerrit/#/c/63963/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/63963/</a> was first stab at making it possible to find it, but that was wrong and won't help in this specific case (because it's async; otherwise it would have, and the NullSafeCollections utility created there has value for other future problems like that IMHO).</p> <p>&gt; Once c/63965 and c/63966 will have been merged, you'll get another <br/> &gt; exception in logs ("Runnnable likely about to terminate thread due to <br/> &gt; uncaught exception; but here is useful debugging context:"),<br/> &gt; please post that here when seen</p> <p>all merged now, so you should soon start seeing the new messsage in CSIT...</p> <p>I'd like to wrap up issuses that are currently assigned to me in JIRA.. normally recent CSIT should include new error messages like "Runnnable likely about to terminate thread due to uncaught exception;" (NB Runnnable is misspelt with 3 instead of 2 n!) - have any of you spotted these meanwhile, or could look out for them? Without hearing back from you within 1 week, I'd like to close this issue.</p> <p>Here is an example that has the log: <a href="https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-ocata-gate-stateful-oxygen/346/odl_1/odl1_karaf.log.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-ocata-gate-stateful-oxygen/346/odl_1/odl1_karaf.log.gz</a></p> <p>What do we do now? I guess the bug is still there but somehow this extra debug points us to the issue?</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>2017-11-17 16:42:45,420 | ERROR | nPool-1-worker-3 | JobCoordinatorImpl | 261 - org.opendaylight.infrautils.jobcoordinator-impl - 1.3.0.SNAPSHOT | Runnnable likely about to terminate thread due to uncaught exception; but here is useful debugging context: JobEntry{key='elaninterface-ff5d93d3-cade-4164-9599-9431c4caa71b', mainWorker=org.opendaylight.netvirt.elan.internal.ElanInstanceManager$$Lambda$768/1055682102@4fc14129, rollbackWorker=null, retryCount=6, futures=[null, com.google.common.util.concurrent.ImmediateFuture$ImmediateSuccessfulCheckedFuture@4bfecc09]} java.lang.NullPointerException: at index 0 2017-11-17 16:42:45,421 | ERROR | nPool-1-worker-3 | JobCoordinatorImpl | 261 - org.opendaylight.infrautils.jobcoordinator-impl - 1.3.0.SNAPSHOT | Thread terminated due to uncaught exception: ForkJoinPool-1-worker-3 java.lang.NullPointerException: at index 0 2017-11-17 16:42:45,421 | ERROR | nPool-1-worker-1 | JobCoordinatorImpl | 261 - org.opendaylight.infrautils.jobcoordinator-impl - 1.3.0.SNAPSHOT | Runnnable likely about to terminate thread due to uncaught exception; but here is useful debugging context: JobEntry{key='elaninterface-c5edf810-4ec7-43a9-9f6b-11a5ff3747b1', mainWorker=org.opendaylight.netvirt.elan.internal.ElanInstanceManager$$Lambda$768/1055682102@12d144e3, rollbackWorker=null, retryCount=6, futures=[null, com.google.common.util.concurrent.ImmediateFuture$ImmediateSuccessfulCheckedFuture@4bfecc09]} java.lang.NullPointerException: at index 0 2017-11-17 16:42:45,421 | ERROR | nPool-1-worker-1 | JobCoordinatorImpl | 261 - org.opendaylight.infrautils.jobcoordinator-impl - 1.3.0.SNAPSHOT | Thread terminated due to uncaught exception: ForkJoinPool-1-worker-1 java.lang.NullPointerException: at index 0 </pre> </div></div> <p>&gt; What do we do now? I guess the bug is still there but somehow this extra debug points us to the issue?</p> <p>yeah, just like I said in the previous comment - did you have a chance to actually read that? <img class="emoticon" src="https://jira.opendaylight.org/images/icons/emoticons/wink.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>The new output now shows that something in ElanInstanceManager submits a DJC job with null future.</p> <p>I'm moving this issue back from In Progress to In Review and will propose a fix when I have time.</p> <p>I've just analysed this one, but I'm not yet getting what's happening here:</p> <p>So the log above shows us that we must have an <tt>enqueueJob</tt> which return a <tt>List&lt;ListenableFuture&lt;Void&gt;</tt> where the 1st of 2 such Future is null, note the clear <tt>futures=null, ...</tt> - and we can see that it's one from a lambda in ElanInstanceManager.</p> <p>However, in ElanInstanceManager there are only 2 such lambdas, first one in line 110 in <tt>remove</tt> which returns 2 futures, first one from elanInterfaceManager.removeElanInterface() in line 90 and second one one directly from <tt>WriteTransaction.submit</tt> in line 118. If we trust that WriteTransaction.submit will never return null (or could that happen?), then it would have to come from <tt>ElanInterfaceManager.removeElanInterface()</tt>, however that in lines 275 &amp; 276 uses ElanUtils.waitForTransactionToComplete which is again just a <tt>WriteTransaction.submit</tt> ? (We'll ignore the fact that this synchronous blocking completely defeats the purpose of using the JC in the first place.)</p> <p>The other such lambda is in line 155 in <tt>update</tt>; that looks wrong, and proposed fix is in <a href="https://git.opendaylight.org/gerrit/#/c/65802/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/65802/</a> - but that cannot lead to this NPE, because it return a null List (which jobcoordinator handles..) and the List with a null element - which we clearly see now in the log.</p> <p>What am I missing?</p> <p><a href="https://git.opendaylight.org/gerrit/65803" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/65803</a> adds explicit null checks to all returned WriteTransaction's submit() JUST in case there is some obscure CDS bug which makes that return null sometimes - but I doubt it. Best I can think of though, so let us retry with that (and c/65802 - JUST in case), and then see again.</p> <p>The futures List in remove is reused across the submitted jobs in the forEach loop:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> DataStoreJobCoordinator dataStoreJobCoordinator = DataStoreJobCoordinator.getInstance(); ElanUtils.removeAndGetElanInterfaces(elanName).forEach(elanInterfaceName -&gt; { dataStoreJobCoordinator.enqueueJob(ElanUtils.getElanInterfaceJobKey(elanInterfaceName), () -&gt; { WriteTransaction writeConfigTxn = broker.newWriteOnlyTransaction(); ... futures.add(writeConfigTxn.submit()); return futures; }, ElanConstants.JOB_MAX_RETRIES); }); </pre> </div></div> <p>This looks very suspicious and unsafe as the List is accessed by multiple threads. I think this is what causes the strange behavior with a null element in the list - tx.submit() never returns null. Each job should return it's own local futures List. I'll submit a patch to address this ...</p> <p>&gt; I'll submit a patch to address this ...</p> <p>==&gt; <a href="https://git.opendaylight.org/gerrit/#/c/65821/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/65821/</a></p> <p>FTR to anyone else re-reviewing this in the future: The important fix is the ditching of the outside <tt>List&lt;ListenableFuture&lt;Void&gt;&gt; futures</tt> usage inside the <tt>forEach</tt> in <tt>remove()</tt>, because that multi-threaded use was badly wrong: Replacing of the <tt>futures.add(writeConfigTxn.submit());</tt> by <tt>return Collections.singletonList(writeConfigTxn.submit());</tt> which is perfectly fine, as the goal is for the JC to handle that list of futures, not to have such a list spanning accross the <tt>forEach</tt>. However the removal of the <tt>futures.addAll()</tt> of what <tt>elanInterfaceManager.removeElanInterface()</tt> returns is a kind of unrelated minor clean up fall out - that part is not what actually fixes the concurrency issue here. I'm just writing this down here because it initially confused me while reviewing it. </p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=shague" class="user-hover" rel="shague">shague</a> can you confirm that you're good with <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=tpantelis" class="user-hover" rel="tpantelis">tpantelis</a> fixing the problem originally raised in this issue - OK to close?</p> <p>PS: Not directly related to this issue, so to be followed-up in separately raised future changes, on staring more at code around this I have to agree with this comment from <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=tpantelis" class="user-hover" rel="tpantelis">tpantelis</a> in the Commit Message of c/65821, quote:</p> <blockquote><p>The futures returned from elanInterfaceManager.removeElanInterface<br/> were also added to the List but I don't see the purpose of this. removeElanInterface<br/> waits for those tx's to complete. Maybe it was so the JC would retry them? If so, this<br/> entire code path has a mix of sync and async transactions which is a bit confusing -<br/> some are submitted to the JC, ohers aren't; some may be retried, others aren't.</p></blockquote> <p>It looks like there is some mix up which makes no sense to me either; I'll propose something about this in 1-2 follow-up changes.</p> <p>&gt; It looks like there is some mix up which makes no sense to me either; I'll propose something about this in 1-2 follow-up changes.</p> <p><a href="https://git.opendaylight.org/gerrit/#/c/65839/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/65839/</a></p> <p>FTR: Something related to this has just come up again in <a href="https://git.opendaylight.org/gerrit/#/c/78678/5/jobcoordinator/jobcoordinator-impl/src/main/java/org/opendaylight/infrautils/jobcoordinator/internal/JobCoordinatorImpl.java@416" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/78678/5/jobcoordinator/jobcoordinator-impl/src/main/java/org/opendaylight/infrautils/jobcoordinator/internal/JobCoordinatorImpl.java@416</a> ...</p> Development External issue ID 9238 External issue URL Issue Type ODL SR Target Milestone Priority Rank 0|i01uvr: