https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/GENIUS-15 genius <p>2016-08-29 08:14:56,152 | ERROR | nPool-1-worker-2 | DataStoreJobCoordinator | 313 - org.opendaylight.genius.mdsalutil-api - 0.2.0.SNAPSHOT | Exception when executing jobEntry: JobEntry</p> {key='org.opendaylight.controller.config.yang.config.legacy_entity_ownership_service_provider.LegacyEntityOwnershipServiceProviderModule$1@50468e80', mainWorker=org.opendaylight.genius.utils.clustering.ClusteringUtils$CheckEntityOwnerTask@19edf62a, rollbackWorker=null, retryCount=0, futures=null} <p>, exception: <span class="error">&#91;org.opendaylight.controller.config.yang.config.legacy_entity_ownership_service_provider.$Proxy52.getOwnershipState(Unknown Source), org.opendaylight.genius.utils.clustering.ClusteringUtils$CheckEntityOwnerTask.call(ClusteringUtils.java:83), org.opendaylight.genius.utils.clustering.ClusteringUtils$CheckEntityOwnerTask.call(ClusteringUtils.java:63), org.opendaylight.genius.datastoreutils.DataStoreJobCoordinator$MainTask.run(DataStoreJobCoordinator.java:248), java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1423), java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289), java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:902), java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1689), java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1644), java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)&#93;</span><br/> java.lang.reflect.UndeclaredThrowableException<br/> at org.opendaylight.controller.config.yang.config.legacy_entity_ownership_service_provider.$Proxy52.getOwnershipState(Unknown Source)<br/> at org.opendaylight.genius.utils.clustering.ClusteringUtils$CheckEntityOwnerTask.call(ClusteringUtils.java:83)<br/> at org.opendaylight.genius.utils.clustering.ClusteringUtils$CheckEntityOwnerTask.call(ClusteringUtils.java:63)<br/> at org.opendaylight.genius.datastoreutils.DataStoreJobCoordinator$MainTask.run(DataStoreJobCoordinator.java:248)<br/> at java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1423)<br/> at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)<br/> at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:902)<br/> at java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1689)<br/> at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1644)<br/> at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)<br/> Caused by: java.lang.reflect.InvocationTargetException<br/> at sun.reflect.GeneratedMethodAccessor118.invoke(Unknown Source)<br/> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)<br/> at java.lang.reflect.Method.invoke(Method.java:483)<br/> at org.opendaylight.controller.config.yang.config.legacy_entity_ownership_service_provider.LegacyEntityOwnershipServiceProviderModule$1.handleInvocation(LegacyEntityOwnershipServiceProviderModule.java:53)<br/> at com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:87)<br/> ... 10 more<br/> Caused by: org.osgi.service.blueprint.container.ServiceUnavailableException: The Blueprint container is being or has been destroyed: (objectClass=org.opendaylight.mdsal.eos.dom.api.DOMEntityOwnershipService)<br/> at org.apache.aries.blueprint.container.ReferenceRecipe.getService(ReferenceRecipe.java:241)<br/> at org.apache.aries.blueprint.container.ReferenceRecipe.access$000(ReferenceRecipe.java:56)<br/> at org.apache.aries.blueprint.container.ReferenceRecipe$ServiceDispatcher.call(ReferenceRecipe.java:306)<br/> at Proxy590a948f_3def_402c_973a_9ad6f6115b8e.getOwnershipState(Unknown Source)<br/> at org.opendaylight.controller.md.sal.dom.clustering.impl.LegacyEntityOwnershipServiceAdapter.getOwnershipState(LegacyEntityOwnershipServiceAdapter.java:60)<br/> at Proxy7b737b0a_4c17_4053_bb8d_a311809db80d.getOwnershipState(Unknown Source)<br/> ... 15 more</p> <p>Operating System: All<br/> Platform: All</p> GENIUS-15

DataStoreJobCoordinator: java.lang.reflect.UndeclaredThrowableException

Bug Resolved Done Michael Vorburger Sam Hague Mon, 29 Aug 2016 12:24:30 +0000 Thu, 6 Apr 2017 15:34:18 +0000 Thu, 6 Apr 2017 15:34:18 +0000 (unspecified) General 0 1 <p>Attachment karaf.tar.gz has been added with description: karaf.log</p> <p>Please specify the steps to reproduce</p> <p>This is captured when using the NetvirtIT. The test simply creates a neutron network and two vm's on the network. Then issues a ping from one vm to the other. To use the IT you need dockr and docker-compose installed on the host. Build the netvirt code and then run the command below. ovsdb.controller.address is the address show for the docker0 interface if you issues ip addr on the host.</p> <p>cd netvirt/vpnservice<br/> mvn -nsu -f it/impl/pom.xml verify -Pintegrationtest -Dovsdb.controller.address=172.17.0.1 -Dit.test=NetvirtIT#testNeutronNet</p> <p>Done a quick first analysis of this... it's some (interesting!) mix up of checked and unchecked exceptions related to not 1 but 2 instances of java.lang.reflect.Proxy, one by Aries BP and one from org.opendaylight.controller.config.yang.config.legacy_entity_ownership_service_provider.LegacyEntityOwnershipServiceProviderModule.createInstance() ...</p> <p>... not sure yet what the right fix is. Also the LegacyEntityOwnershipServiceProviderModule is from some pre-BP pure CSS times, should ideally be removed, but let's try to fix it as is first. The actual bug and thus fix may be in Controller, not Genius (I'll move the Bugzilla Product if I'm sure).</p> <p>&gt; Caused by: org.osgi.service.blueprint.container.ServiceUnavailableException: <br/> &gt; The Blueprint container is being or has been destroyed:</p> <p>Sam, have you seen this one "in the wild" (during real operations) or only during NetvirtIT ? I think this MAY (I'm not sure) be one of our (too many... yes) problems happening (only) during shutdown / feature un-install? It certainly would be good to fix it either way, but the priority would be a little different.</p> <p>&gt; one of our (too many... yes) problems happening (only)<br/> &gt; during shutdown / feature un-install</p> <p>actually, even if I "fix" this (the Exception above), then another one will ust appear.. the real issue here probably is that a background thread in DataStoreJobCoordinator is still trying to access EntityOwnershipService when that bundles' "Blueprint container is being or has been destroyed" - Karaf is already shutting down.</p> <p>The real fix here then is EITHER to stop DataStoreJobCoordinator correctly, which I've proposed in <a href="https://git.opendaylight.org/gerrit/#/c/52976/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/52976/</a> (that's the right generic solution IMHO if whatever that background job does actually isn't really important), OR for maybe for NetvirtIT to wait for the DataStoreJobCoordinator to be done (that's the right local fix if the NetvirtIT does care about the outcome of that background job - but then it really should be extended with an assert of something that happens in that job?).</p> <p><a href="https://git.opendaylight.org/gerrit/#/c/53412/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/53412/</a></p> <p>will this resolve the problem?</p> <p><a href="https://git.opendaylight.org/gerrit/#/c/53412/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/53412/</a></p> <p>&gt; will c/53412 resolve the problem?</p> <p>with this now merged, I expect another exception showing the actual root cause will probably appear now for this (not tested; perhaps Sam or Faseela you could copy/paste the new exception if you see it happen?).</p> <p>The real issue here is that a background thread in DataStoreJobCoordinator is still trying to access EntityOwnershipService when that bundles' "Blueprint container is being or has been destroyed" - Karaf is already shutting down.</p> <p>The real fix here then is EITHER to stop DataStoreJobCoordinator correctly, which I've proposed in <a href="https://git.opendaylight.org/gerrit/#/c/52976/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/52976/</a> (that's the right generic solution IMHO if whatever that background job does actually isn't really important), OR for maybe for NetvirtIT to wait for the DataStoreJobCoordinator to be done (that's the right local fix if the NetvirtIT does care about the outcome of that background job - but then it really should be extended with an assert of something that happens in that job?).</p> <p><a href="https://git.opendaylight.org/gerrit/#/c/52976/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/52976/</a> is now also merged, so the error above won't occur anymore like that. &#8211; It's possible that another exception will probably appear now at the end of the NetvirtIT (not tested), but it will look very different, so let's track that in a new bug to be opened when someone hits it.</p> <p>PS: Maybe NetvirtIT would have to wait for the DataStoreJobCoordinator to be done (that's the right local fix if the NetvirtIT does care about the outcome of that background job - but then it really should be extended with an assert of something that happens in that job?).</p> Development External issue ID 6564 External issue URL Rank 0|i01obr: