https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/CONTROLLER-893 controller <p>System tests for VTN frequently fails.</p> <p><a href="https://jenkins.opendaylight.org/integration/job/integration-master-csit-karaf-vtn-only/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/integration/job/integration-master-csit-karaf-vtn-only/</a><br/> <a href="https://jenkins.opendaylight.org/integration/job/integration-master-csit-karaf-vtn-all/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/integration/job/integration-master-csit-karaf-vtn-all/</a></p> <p>We've found out that the root cause of this failure is in clustering.</p> <p>We sometimes observe that clustering.services-implementation fails to register listener.<br/> Which listener the clustering.services-implementation fails to register changes each time.</p> <p>We observed that when the clustering.services-implementation fails to register topologymanager.edgesDB, system tests for VTN failed.</p> <p> 2014-09-24 14:29:00,198 | TRACE | Local user karaf | \<br/> ClusterGlobalManager | \<br/> 98 - org.opendaylight.controller.clustering.services-implementation - \<br/> 0.4.3.Helium | setCacheUpdateAware: \<br/> org.opendaylight.controller.topologymanager.internal.\<br/> TopologyManagerImpl@ceb60d</p> <p> 2014-09-24 14:29:00,199 | TRACE | Local user karaf | \<br/> ClusterManagerCommon | \<br/> 98 - org.opendaylight.controller.clustering.services-implementation - \<br/> 0.4.3.Helium | CacheUpdateAware being set on container:default</p> <p> 2014-09-24 14:29:00,199 | TRACE | Local user karaf | \<br/> ClusterManagerCommon | \<br/> 98 - org.opendaylight.controller.clustering.services-implementation - \<br/> 0.4.3.Helium | cachenames provided below:</p> <p> 2014-09-24 14:29:00,199 | DEBUG | Local user karaf | \<br/> ClusterManagerCommon | \<br/> 98 - org.opendaylight.controller.clustering.services-implementation - \<br/> 0.4.3.Helium | Cache topologymanager.edgesDB didn't exist when \<br/> org.opendaylight.controller.topologymanager.internal.\<br/> TopologyManagerImpl@ceb60d tried to register to its updates</p> <p>Because if the clustering.services-implementation fails to register topologymanager.edgesDB, dijkstra_impelmentation fails to create route information, and IRouting.getRoute(Node, Node) returns null, VTN Manager thinks there are no route between hosts, and eventually VTN Manager does not try to forward packets and install flow entries.</p> <p>That is why System tests for VTN frequently fails.</p> <p>Operating System: All<br/> Platform: All</p> CONTROLLER-893

clustering.services-implementation failed to register listner.

Bug Resolved Done Unassigned Hideyuki Tai Thu, 25 Sep 2014 00:25:14 +0000 Thu, 19 Oct 2017 22:13:23 +0000 Mon, 29 Sep 2014 06:47:56 +0000 Helium clustering 0 6 <p>The easiest way to reproduce the issue<br/> ======================================</p> <p> 0. Clean caches.</p> <p> $ rm -rf data/</p> <p> 1. Run Karaf distribution.</p> <p> $ bin/karaf</p> <p> 2. Change the log level of clustering.services_implementation to TRACE</p> <p> log:set TRACE org.opendaylight.controller.clustering.services_implementation</p> <p> 3. Install odl-nsf-managers</p> <p> feature:install odl-nsf-managers</p> <p> NOTE: Which listener the clustering.services-implementation fails to register changes each time.</p> <p>My observation<br/> ==============</p> <p>The following is my observation during the above procedure.</p> {nodeconnections] It was failed to register listener for nodeconnections. 2014-09-24 20:41:51,777 | DEBUG | Local user karaf | ClusterManagerCommon | 189 - org.opendaylight.controller.clustering.services-implementation - 0.4.3.Helium-RC2-test-v201409232153 | Cache connectionmanager.CONTAINER_BASED.nodeconnections didn't exist when org.opendaylight.controller.connectionmanager.internal.ConnectionManager@34455f88 tried to register to its updates 2014-09-24 20:41:51,777 | DEBUG | Local user karaf | ClusterManagerCommon | 189 - org.opendaylight.controller.clustering.services-implementation - 0.4.3.Helium-RC2-test-v201409232153 | Cache connectionmanager.LOAD_BALANCED.nodeconnections didn't exist when org.opendaylight.controller.connectionmanager.internal.ConnectionManager@34455f88 tried to register to its updates 2014-09-24 20:41:51,778 | DEBUG | Local user karaf | ClusterManagerCommon | 189 - org.opendaylight.controller.clustering.services-implementation - 0.4.3.Helium-RC2-test-v201409232153 | Cache connectionmanager.ROUND_ROBIN.nodeconnections didn't exist when org.opendaylight.controller.connectionmanager.internal.ConnectionManager@34455f88 tried to register to its u pdates [config.container.event.save] 2014-09-24 20:41:51,926 | DEBUG | Local user karaf | ClusterManagerCommon | 189 - org.opendaylight.controller.clustering.services-implementation - 0.4.3.Helium-RC2-test-v201409232153 | Cache config.container.event.save didn't exist when org.opendaylight.controller.configuration.internal.ContainerConfigurationService@408c3e7b tried to register to its updates [statisticsmanager] 2014-09-24 20:41:51,945 | DEBUG | Local user karaf | ClusterManagerCommon | 189 - org.opendaylight.controller.clustering.services-implementation - 0.4.3.Helium-RC2-test-v201409232153 | Cache statisticsmanager.triggers didn't exist when org.opendaylight.controller.statisticsmanager.internal.StatisticsManager@53f21293 tried to register to its updates 2014-09-24 20:41:51,945 | DEBUG | Local user karaf | ClusterManagerCommon | 189 - org.opendaylight.controller.clustering.services-implementation - 0.4.3.Helium-RC2-test-v201409232153 | Cache statisticsmanager.flowStatistics didn't exist when org.opendaylight.controller.statisticsmanager.internal.StatisticsManager@53f21293 tried to register to its updates [frm.workStatus] 2014-09-24 20:41:52,029 | DEBUG | Local user karaf | ClusterManagerCommon | 189 - org.opendaylight.controller.clustering.services-implementation - 0.4.3.Helium-RC2-test-v201409232153 | Cache frm.workOrder didn't exist when org.opendaylight.controller.forwardingrulesmanager.internal.ForwardingRulesManager@3ce3ed23 tried to register to its updates 2014-09-24 20:41:52,029 | DEBUG | Local user karaf | ClusterManagerCommon | 189 - org.opendaylight.controller.clustering.services-implementation - 0.4.3.Helium-RC2-test-v201409232153 | Cache frm.workStatus didn't exist when org.opendaylight.controller.forwardingrulesmanager.internal.ForwardingRulesManager@3ce3ed23 tried to register to its updates 2014-09-24 20:41:52,030 | DEBUG | Local user karaf | ClusterManagerCommon | 189 - org.opendaylight.controller.clustering.services-implementation - 0.4.3.Helium-RC2-test-v201409232153 | Cache frm.installedSwView didn't exist when org.opendaylight.controller.forwardingrulesmanager.internal.ForwardingRulesManager@3ce3ed23 tried to register to its updates {topologymanager.edgesDB} <p>2014-09-24 20:41:52,041 | DEBUG | Local user karaf | ClusterManagerCommon | 189 - org.opendaylight.controller.clustering.services-implementation - 0.4.3.Helium-RC2-test-v201409232153 | Cache topologymanager.edgesDB didn't exist when org.opendaylight.controller.topologymanager.internal.TopologyManagerImpl@7f6d2f18 tried to register to its updates</p> <p>Adding Moiz to the cc list in the hope that he can take this or find somebody who can.</p> <p>Hideyuki... </p> <p>Please try to collect the logs corresponding to this:</p> <p> private void retrieveCaches() {<br/> if (this.clusterContainerService == null) </p> { log.error("Cluster Services is null, can't retrieve caches."); return; } <p> this.edgesDB = (ConcurrentMap&lt;Edge, Set&lt;Property&gt;&gt;) this.clusterContainerService.getCache(TOPOEDGESDB);<br/> if (edgesDB == null) </p> { log.error("Failed to get cache for " + TOPOEDGESDB); } <p> this.hostsDB =<br/> (ConcurrentMap&lt;NodeConnector, Set&lt;ImmutablePair&lt;Host, Set&lt;Property&gt;&gt;&gt;&gt;) this.clusterContainerService.getCache(TOPOHOSTSDB);<br/> if (hostsDB == null) </p> { log.error("Failed to get cache for " + TOPOHOSTSDB); } <p> this.nodeConnectorsDB =<br/> (ConcurrentMap&lt;NodeConnector, Set&lt;Property&gt;&gt;) this.clusterContainerService.getCache(TOPONODECONNECTORDB);<br/> if (nodeConnectorsDB == null) </p> { log.error("Failed to get cache for " + TOPONODECONNECTORDB); } <p> this.userLinksDB =<br/> (ConcurrentMap&lt;String, TopologyUserLinkConfig&gt;) this.clusterContainerService.getCache(TOPOUSERLINKSDB);<br/> if (userLinksDB == null) </p> { log.error("Failed to get cache for " + TOPOUSERLINKSDB); } <p> }</p> <p>From TopologyManagerImpl.</p> <p>The caches being reported missing are created and retrieved in TopologyManager.init().</p> <p>I think we should see:<br/> ClusterServicesCommons.init()<br/> TopologyManagerImpl.init()<br/> and <b>then</b> the injection of the TopologyManagerImpl as IClusterAware into the ClusterServicesCommons</p> <p>So I am hoping we can see from the logs whether we are seeing errors registering the caches.</p> <p>Somebody might try a retry with a sleep in setClusterAware in ClusterServicesCommon. Then make sure it doesn't slow down boot unacceptably.</p> <p>Other dirty solutions would be:</p> <p>1.) Telling users how to tell if VTN came up good or not, so they can reboot if not.<br/> 2.) Writing a script to automate 1.</p> <p>I tried reproducing this issue.</p> <p>Couple of things,</p> <p>1. I do not see this message logged at all</p> <p>logger.trace("cachename:{} on container:{} has " +<br/> "been registered", cache,<br/> this.containerName);</p> <p>2. Where the exception related message is logged I see the comment </p> <p>// Do nothing, the important is that<br/> // we don't register the listener in<br/> // the shadow, and we are not doing<br/> // that.</p> <p>So it seems to me that (a) caches never exist and (b) For some reason it is ok to ignore the adding of the listener</p> <p>Ed, None of the errors that you asked Hideyuki to look for are reported. There was one though</p> <p>2014-09-26 09:21:19,868 | ERROR | Local user karaf | ConfigurationService | 195 - org.opendaylight.controller.configuration.implementation - 0.4.3.SNAPSHOT | Failed to create config directory: configuration/startup/default</p> <p>I don't know if this has any bearing on the issue.</p> <p>I tried adding 100 retires when adding a listener with a sleep of upto 100 millisecond between retries and it still failed when adding many listeners.</p> <p>I added some additional logging in createCache to see if the cache was ever created. It looks like when the add listener fails the cache is never created.</p> <p>(In reply to Moiz Raja from comment #8)<br/> &gt; I added some additional logging in createCache to see if the cache was ever<br/> &gt; created. It looks like when the add listener fails the cache is never<br/> &gt; created.</p> <p>The cache is created afterward, so the idea is actually to store those caches that has been got listener before the creation and then to add the listener later on.</p> <p>Giovanni,</p> <p>I added logging like so,</p> <p> @Override<br/> public ConcurrentMap&lt;?, ?&gt; createCache(String cacheName,<br/> Set&lt;IClusterServices.cacheMode&gt; cMode) throws CacheExistException,<br/> CacheConfigException {<br/> if (this.clusterService != null) {</p> <p> logger.info("Creating cache {} success!!!!", cacheName);<br/> return this.clusterService.createCache(this.containerName,<br/> cacheName, cMode);<br/> } else {<br/> logger.error("Could not create cache {} because clusterService is null", cacheName);<br/> return null;<br/> }<br/> }</p> <p>and I see neither of the log statements in the log.</p> <p>Attachment karaf.log has been added with description: karaf.log showing that create cache was not called for caches where adding listener failed</p> <p>(In reply to Moiz Raja from comment #11)<br/> &gt; Created attachment 266 <span class="error">&#91;details&#93;</span><br/> &gt; karaf.log showing that create cache was not called for caches where adding<br/> &gt; listener failed</p> <p>Thanks Moiz, but this logs show no caches got the message "didn't exist when", so not sure if the log is telling it all.</p> <p>Here are the other changes I did. I did comment out the "didn't exist" statement.</p> <p> void setCacheUpdateAware(Map props, ICacheUpdateAware s) {<br/> logger.info("MOIZ: CacheUpdateAware being set on container:{}",<br/> this.containerName);</p> <p> if (this.cacheUpdateAware != null) {<br/> Set&lt;String&gt; caches = (Set&lt;String&gt;)props.get("cachenames");<br/> if (caches != null) {<br/> logger.trace("cachenames provided below:");<br/> for (String cache : caches) {<br/> if (this.cacheUpdateAware.get(cache) != null) {<br/> logger.error("cachename:{} on container:{} has already a listener", cache, this.containerName);<br/> } else {<br/> GetUpdatesContainer&lt;?, ?&gt; up = new GetUpdatesContainer(s, this.containerName, cache);<br/> if (up != null) </p> { addListener(s, cache, up); } <p> }<br/> }<br/> }<br/> }<br/> }</p> <p> private void addListener(ICacheUpdateAware s, String cache, GetUpdatesContainer&lt;?, ?&gt; up) {</p> <p> int tries;</p> <p> int MAX_RETRIES = 5;</p> <p> for(tries=0;tries&lt;MAX_RETRIES;tries++) {<br/> try {<br/> this.clusterService.addListener(this.containerName,<br/> cache, up);<br/> this.cacheUpdateAware.put(cache, up);<br/> logger.info("cachename:{} on container:{} has " +<br/> "been registered", cache,<br/> this.containerName<br/> );</p> <p> break;<br/> } catch (CacheListenerAddException exc) {<br/> // logger<br/> // .debug("Cache {} didn't exist when {} tried to register to its updates, tries = {}", cache,<br/> // s, tries);<br/> // Do nothing, the important is that<br/> // we don't register the listener in<br/> // the shadow, and we are not doing<br/> // that.<br/> }</p> <p> try </p> { Thread.sleep(100); } <p> catch (InterruptedException e) {<br/> }<br/> }</p> <p> if(tries == MAX_RETRIES){<br/> logger.error("Failed to add listener for cache {} after {} tries", cache, MAX_RETRIES);<br/> }<br/> }</p> <p>Moiz Raja has written a patch for this bug.</p> <p><a href="https://git.opendaylight.org/gerrit/#/c/11621/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/11621/</a></p> <p>Hiedyuki, if you have confirmed that this is now fixed, can you set the state to fixed, resolved?</p> <p>Thanks!</p> Development External issue ID 2074 External issue URL Rank 0|i02n7j: