https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/CONTROLLER-1518 controller <p>We have observed that DataChangeListener was not notified of data change occasionally.<br/> This issue happens on the master branch (for Boron development).</p> <p>When the issue happened, we observed the following exception was caused during DS update.</p> <p>2016-05-16 15:17:21,371 | ERROR | lt-dispatcher-32 | OneForOneStrategy | 184 - com.typesafe.akka.slf4j - 2.4.4 | This stopwatch is already running.<br/> java.lang.IllegalStateException: This stopwatch is already running.<br/> at com.google.common.base.Preconditions.checkState(Preconditions.java:173)<span class="error">&#91;83:com.google.guava:18.0.0&#93;</span><br/> at com.google.common.base.Stopwatch.start(Stopwatch.java:160)<span class="error">&#91;83:com.google.guava:18.0.0&#93;</span><br/> at org.opendaylight.controller.cluster.datastore.DefaultShardDataChangeListenerPublisher.publishChanges(DefaultShardDataChangeListenerPublisher.java:59)<span class="error">&#91;199:org.opendaylight.controller.sal-distributed-datastore:1.4.0.SNAPSHOT&#93;</span><br/> at org.opendaylight.controller.cluster.datastore.ShardDataTreeNotificationPublisherActor$PublishNotifications.publish(ShardDataTreeNotificationPublisherActor.java:46)<span class="error">&#91;199:org.opendaylight.controller.sal-distributed-datastore:1.4.0.SNAPSHOT&#93;</span><br/> at org.opendaylight.controller.cluster.datastore.ShardDataTreeNotificationPublisherActor$PublishNotifications.access$000(ShardDataTreeNotificationPublisherActor.java:33)<span class="error">&#91;199:org.opendaylight.controller.sal-distributed-datastore:1.4.0.SNAPSHOT&#93;</span><br/> at org.opendaylight.controller.cluster.datastore.ShardDataTreeNotificationPublisherActor.handleReceive(ShardDataTreeNotificationPublisherActor.java:25)<span class="error">&#91;199:org.opendaylight.controller.sal-distributed-datastore:1.4.0.SNAPSHOT&#93;</span><br/> at org.opendaylight.controller.cluster.common.actor.AbstractUntypedActor.onReceive(AbstractUntypedActor.java:25)<span class="error">&#91;195:org.opendaylight.controller.sal-clustering-commons:1.4.0.SNAPSHOT&#93;</span><br/> at akka.actor.UntypedActor$$anonfun$receive$1.applyOrElse(UntypedActor.scala:165)<span class="error">&#91;183:com.typesafe.akka.actor:2.4.4&#93;</span><br/> at akka.actor.Actor$class.aroundReceive(Actor.scala:482)<span class="error">&#91;183:com.typesafe.akka.actor:2.4.4&#93;</span><br/> at akka.actor.UntypedActor.aroundReceive(UntypedActor.scala:95)<span class="error">&#91;183:com.typesafe.akka.actor:2.4.4&#93;</span><br/> at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)<span class="error">&#91;183:com.typesafe.akka.actor:2.4.4&#93;</span><br/> at akka.actor.ActorCell.invoke(ActorCell.scala:495)<span class="error">&#91;183:com.typesafe.akka.actor:2.4.4&#93;</span><br/> at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)<span class="error">&#91;183:com.typesafe.akka.actor:2.4.4&#93;</span><br/> at akka.dispatch.Mailbox.run(Mailbox.scala:224)<span class="error">&#91;183:com.typesafe.akka.actor:2.4.4&#93;</span><br/> at akka.dispatch.Mailbox.exec(Mailbox.scala:234)<span class="error">&#91;183:com.typesafe.akka.actor:2.4.4&#93;</span><br/> at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)<span class="error">&#91;179:org.scala-lang.scala-library:2.11.8.v20160304-115712-1706a37eb8&#93;</span><br/> at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)<span class="error">&#91;179:org.scala-lang.scala-library:2.11.8.v20160304-115712-1706a37eb8&#93;</span><br/> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)<span class="error">&#91;179:org.scala-lang.scala-library:2.11.8.v20160304-115712-1706a37eb8&#93;</span><br/> at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)<span class="error">&#91;179:org.scala-lang.scala-library:2.11.8.v20160304-115712-1706a37eb8&#93;</span></p> <p>Due to this issue, IT for VTN Manager on the master branch fails occasionally.</p> <p><a href="https://jenkins.opendaylight.org/releng/view/vtn/job/vtn-verify-boron/138/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/vtn/job/vtn-verify-boron/138/</a><br/> <a href="https://jenkins.opendaylight.org/releng/view/vtn/job/vtn-verify-boron/129/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/vtn/job/vtn-verify-boron/129/</a></p> <p>Operating System: All<br/> Platform: All</p> CONTROLLER-1518

DataChangeListener is not notified of DS change occasionally

Bug Resolved Done Unassigned Hideyuki Tai Wed, 18 May 2016 23:00:26 +0000 Thu, 19 Oct 2017 21:25:25 +0000 Wed, 15 Jun 2016 04:00:20 +0000 0.4.0 clustering 0 3 <p>We've investigated this issue.<br/> We are thinking that the root cause of this issue is in the <b>sal-distributed-datastore</b>.</p> <p>When VTN project used <b>sal-inmemory-datastore</b> as the backend of DS, VTN project has never faced this issue. However, after VTN project migrated it's IT into mdsal-it-base, the VTN IT uses the <b>sal-distributed-datastore</b>, and the issue has showed up.</p> <p>What happened when the IllegalStateException occurred?<br/> In the failure case, the publishChanges() of DefaultShardDataChangeListenerPublisher class was called from multiple threads at the same time.<br/> Then, Stopwatch.start() was called even though the stopwatch was already started on a different thread for the same instance of Stopwatch class.</p> <p><span class="error">&#91;opendaylight/md-sal/sal-distributed-datastore/src/main/java/org/opendaylight/controller/cluster/datastore/DefaultShardDataChangeListenerPublisher.java&#93;</span></p> <p> 33 final class DefaultShardDataChangeListenerPublisher implements ShardDataChangeListenerPublisher,<br/> 34 NotificationManager&lt;DataChangeListenerRegistration&lt;?&gt;, DOMImmutableDataChangeEvent&gt; {<br/> (snip)</p> <p> 38 private final Stopwatch timer = Stopwatch.createUnstarted();</p> <p> (snip)</p> <p> 57 @Override<br/> 58 public void publishChanges(DataTreeCandidate candidate, String logContext) {<br/> 59 timer.start();<br/> 60<br/> 61 try </p> { 62 ResolveDataChangeEventsTask.create(candidate, dataChangeListenerTree).resolve(this); 63 } <p> finally {<br/> 64 timer.stop();<br/> 65 long elapsedTime = timer.elapsed(TimeUnit.MILLISECONDS);<br/> 66 if(elapsedTime &gt;= PUBLISH_DELAY_THRESHOLD_IN_MS) {<br/> 67 LOG.warn("{}: Generation of DataChange events took longer than expected. Elapsed time: {}",<br/> 68 logContext, timer);<br/> 69 } else {<br/> 70 LOG.debug("{}: Elapsed time for generation of DataChange events: {}", logContext, timer);<br/> 71 }<br/> 72<br/> 73 timer.reset();<br/> 74 }<br/> 75 }</p> <p>Here, we are talking about DefaultShardDataChangeListenerPublisher class so far.<br/> However, we think the DefaultShardDataTreeChangeListenerPublisher has also the same issue.</p> <p>IT for VTN Manager on the master branch still failed on 06/11/2016 due to the <a href="https://jira.opendaylight.org/browse/CONTROLLER-1518" title="DataChangeListener is not notified of DS change occasionally" class="issue-link" data-issue-key="CONTROLLER-1518"><del>CONTROLLER-1518</del></a>.<br/> <a href="https://jenkins.opendaylight.org/releng/job/vtn-integration-boron/896/org.opendaylight.vtn$manager.it.core/console" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/job/vtn-integration-boron/896/org.opendaylight.vtn$manager.it.core/console</a></p> <p>Submitted <a href="https://git.opendaylight.org/gerrit/#/c/40306/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/40306/</a></p> <p>Be patch: <a href="https://git.opendaylight.org/gerrit/#/c/40324/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/40324/</a></p> Development External issue ID 5913 External issue URL Issue Type ODL SR Target Milestone Rank 0|i02r2f: