https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/CONTROLLER-1551 controller <p>ConcurrentModificationException observed when installing bgpcep (master branch)</p> <p>steps to reproduce it:<br/> -run Karaf<br/> -install -&gt; feature:install odl-restconf odl-bgpcep-bgp</p> <p>2016-09-22 16:32:50,282 | WARN | r - ConfigPusher | FeatureConfigPusher | 137 - config-persister-feature-adapter - 0.6.0.SNAPSHOT | Karaf featuresService.listInstalledFeatures() has thrown an exception, retry 0<br/> java.util.ConcurrentModificationException<br/> at java.util.HashMap$HashIterator.nextNode(HashMap.java:1437)<span class="error">&#91;:1.8.0_101&#93;</span><br/> at java.util.HashMap$KeyIterator.next(HashMap.java:1461)<span class="error">&#91;:1.8.0_101&#93;</span><br/> at java.util.AbstractCollection.toArray(AbstractCollection.java:196)<span class="error">&#91;:1.8.0_101&#93;</span><br/> at org.apache.karaf.features.internal.FeaturesServiceImpl.listInstalledFeatures(FeaturesServiceImpl.java:840)<span class="error">&#91;20:org.apache.karaf.features.core:3.0.8&#93;</span><br/> at Proxy9faf1ce0_0954_4724_860e_acc04c402866.listInstalledFeatures(Unknown Source)<span class="error">&#91;:&#93;</span><br/> at org.opendaylight.controller.configpusherfeature.internal.FeatureConfigPusher.isInstalled(FeatureConfigPusher.java:96)<span class="error">&#91;137:config-persister-feature-adapter:0.6.0.SNAPSHOT&#93;</span><br/> at org.opendaylight.controller.configpusherfeature.internal.FeatureConfigPusher.pushConfig(FeatureConfigPusher.java:77)<span class="error">&#91;137:config-persister-feature-adapter:0.6.0.SNAPSHOT&#93;</span><br/> at org.opendaylight.controller.configpusherfeature.internal.FeatureConfigPusher.pushConfigs(FeatureConfigPusher.java:67)<span class="error">&#91;137:config-persister-feature-adapter:0.6.0.SNAPSHOT&#93;</span><br/> at org.opendaylight.controller.configpusherfeature.internal.ConfigPushingRunnable.processFeatureEvent(ConfigPushingRunnable.java:65)<span class="error">&#91;137:config-persister-feature-adapter:0.6.0.SNAPSHOT&#93;</span><br/> at org.opendaylight.controller.configpusherfeature.internal.ConfigPushingRunnable.run(ConfigPushingRunnable.java:47)<span class="error">&#91;137:config-persister-feature-adapter:0.6.0.SNAPSHOT&#93;</span><br/> at java.lang.Thread.run(Thread.java:745)<span class="error">&#91;:1.8.0_101&#93;</span></p> <p>Operating System: All<br/> Platform: All</p> CONTROLLER-1551

ConcurrentModificationException at karaf.FeaturesServiceImpl at opendaylight.controller.configpusherfeature...FeatureConfigPusher

Bug Resolved Done Unassigned Claudio David Gasparini Thu, 22 Sep 2016 14:51:38 +0000 Tue, 25 Jul 2023 08:24:13 +0000 Wed, 5 Jul 2017 13:05:51 +0000 config 0 3 <p>karaf.log</p> <p>Attachment karaf.log has been added with description: karaf.log</p> <p>ConcurrentModificationException observed when installing odl features like odl-restconf which trigers installation of other dependent odl features.<br/> you can reproduce it on latest Carbon SNAPSHOT, it is there for a while.<br/> karaf 3.0.8</p> <p>2017-01-24 22:04:02,935 | INFO | l for user karaf | DataObjectToJsonConverter | 338 - tech.pantheon.visibilitypackage.topologyutils.vp-topology-utils - 0.1.0.SNAPSHOT | Mapping service built<br/> 2017-01-24 22:04:02,951 | INFO | tAdminThread #19 | BlueprintBundleTracker | 176 - org.opendaylight.controller.blueprint - 0.6.0.SNAPSHOT | Blueprint container for bundle tech.pantheon.visibilitypackage.topologygenerator.vp-topology-generator-impl_0.1.0.SNAPSHOT <span class="error">&#91;340&#93;</span> was successfully created<br/> 2017-01-24 22:04:02,953 | WARN | r - ConfigPusher | FeatureConfigPusher | 136 - config-persister-feature-adapter - 0.6.0.SNAPSHOT | Karaf featuresService.listInstalledFeatures() has thrown an exception, retry 0<br/> java.util.ConcurrentModificationException<br/> at java.util.HashMap$HashIterator.nextNode(HashMap.java:1437)<span class="error">&#91;:1.8.0_111&#93;</span><br/> at java.util.HashMap$KeyIterator.next(HashMap.java:1461)<span class="error">&#91;:1.8.0_111&#93;</span><br/> at java.util.AbstractCollection.toArray(AbstractCollection.java:196)<span class="error">&#91;:1.8.0_111&#93;</span><br/> at org.apache.karaf.features.internal.FeaturesServiceImpl.listInstalledFeatures(FeaturesServiceImpl.java:840)<span class="error">&#91;20:org.apache.karaf.features.core:3.0.8&#93;</span><br/> at Proxy6c30fc01_a81f_4a46_a011_c7ae5d9b78e7.listInstalledFeatures(Unknown Source)<span class="error">&#91;:&#93;</span><br/> at org.opendaylight.controller.configpusherfeature.internal.FeatureConfigPusher.isInstalled(FeatureConfigPusher.java:97)<span class="error">&#91;136:config-persister-feature-adapter:0.6.0.SNAPSHOT&#93;</span><br/> at org.opendaylight.controller.configpusherfeature.internal.FeatureConfigPusher.pushConfig(FeatureConfigPusher.java:78)<span class="error">&#91;136:config-persister-feature-adapter:0.6.0.SNAPSHOT&#93;</span><br/> at org.opendaylight.controller.configpusherfeature.internal.FeatureConfigPusher.pushConfigs(FeatureConfigPusher.java:68)<span class="error">&#91;136:config-persister-feature-adapter:0.6.0.SNAPSHOT&#93;</span><br/> at org.opendaylight.controller.configpusherfeature.internal.ConfigPushingRunnable.processFeatureEvent(ConfigPushingRunnable.java:65)<span class="error">&#91;136:config-persister-feature-adapter:0.6.0.SNAPSHOT&#93;</span><br/> at org.opendaylight.controller.configpusherfeature.internal.ConfigPushingRunnable.run(ConfigPushingRunnable.java:47)<span class="error">&#91;136:config-persister-feature-adapter:0.6.0.SNAPSHOT&#93;</span><br/> at java.lang.Thread.run(Thread.java:745)<span class="error">&#91;:1.8.0_111&#93;</span><br/> 2017-01-24 22:04:06,837 | INFO | config-pusher | ConfigPusherImpl | 135 - org.opendaylight.controller.config-persister-impl - 0.6.0.SNAPSHOT | Pushing configuration snapshot 00-netty.xml(vp-all,vp-all)<br/> 2017-01-24 22:04:07,143 | INFO | config-pusher | ConfigPusherImpl | 135 - org.opendaylight.controller.config-persister-impl - 0.6.0.SNAPSHOT | Successfully pushed configuration snapshot 00-netty.xml(vp-all,vp-a</p> <p>Just saw this as well (in netvirt/vpnservice/distribution/karaf on feature:install odl-netvirt-openstack; not related to actual feature being installed) .. not reliably reproducible, as these kinds of bugs typically go, of course.</p> <p>I'm wondering if this is missing synchronization in Karaf's FeaturesServiceImpl or ODL's FeatureConfigPusher? If Karaf's FeaturesService is not meant to be thread safe, but if FeatureConfigPusher can run concurrently (for config of different bundles?), then we probably should ourselves synchronize when we use it? That's not that hard to do... I'll see if I can propose a Gerrit.</p> <p><a href="https://git.opendaylight.org/gerrit/#/c/53341/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/53341/</a></p> <p>In the 3.0.8 version of FeaturesServiceImpl, the 'installed' Map isn't synchronized. The master version at <a href="https://github.com/apache/karaf/blob/master/features/core/src/main/java/org/apache/karaf/features/internal/service/FeaturesServiceImpl.java" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/apache/karaf/blob/master/features/core/src/main/java/org/apache/karaf/features/internal/service/FeaturesServiceImpl.java</a> appears to be correctly synchronized/thread-safe.</p> <p>&gt; the 3.0.8 version of FeaturesServiceImpl (...) The master version</p> <p>OK, then let's say that properly resolving this bug depends on Karaf 4 upgrade <a href="https://jira.opendaylight.org/browse/ODLPARENT-23" title="Milestone: Upgrade karaf to 4.0.1 or later" class="issue-link" data-issue-key="ODLPARENT-23"><del>ODLPARENT-23</del></a>.</p> <p>I'll try to adjust c/53341 with an ugly workaround hack in the mean time..</p> <p>Oh... hang on, small confusion here.. the code in FeatureConfigPusher actually already does handle this with a retry logic! BUT: </p> <p>(a) As three people independently reported this, not understanding that this WARN with the "retry 0" at the end does not actually indicate a problem, including current 2 controller committers not being aware of this, I would suggest that it is wrong to print a WARN log with a scary looking exception if it's "just" retrying - there should be no log (or only a DEBUG) - but an ERROR if it fails after retrying. </p> <p>(b) On looking at this code more closely, I've also found a 2nd place where I guess this could happen and should be handled - in FeatureConfigPusher's pushConfig using featuresService.getFeature. </p> <p>I'll change the Gerrit to do both (a) and (b) ...</p> <p>This, from the current implementation, also seems a bit stupid:</p> <p> private static final int MAX_RETRIES = 100;<br/> private static final int RETRY_PAUSE_MILLIS = 1;</p> <p>I'm going to change it to:</p> <p> private static final int MAX_RETRIES = 10;<br/> private static final int RETRY_PAUSE_MILLIS = 100;</p> <p>&gt; (b) On looking at this code more closely, I've also found a 2nd place where I &gt; guess this could happen and should be handled - in FeatureConfigPusher's <br/> &gt; pushConfig using featuresService.getFeature. </p> <p>I've ignored this, for now; we'll handle it some other time, maybe.</p> <p>PS: Do we have a generic Retryer utility?</p> <p>&gt; PS: Do we have a generic Retryer utility?</p> <p>org.opendaylight.genius.datastoreutils.TaskRetryLooper</p> <p>I believe this was fixed a few months ago - also shouldn't be an issue with karaf 4.</p> <p>&gt; I believe this was fixed a few months ago</p> <p>yes; <a href="https://git.opendaylight.org/gerrit/#/c/53341/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/53341/</a> removed the confusing WARN log.</p> Blocks ODLPARENT-23 Development External issue ID 6787 External issue URL Rank 0|i02r9r: