Uploaded image for project: 'controller'
  1. controller
  2. CONTROLLER-2026

Circuit breaker timeout

    XMLWordPrintable

Details

    • Bug
    • Status: Open
    • High
    • Resolution: Unresolved
    • 4.0.7
    • None
    • clustering
    • None

    Description

      OpenFlow Plugin longevity test, which generates a lot of updates in the inventory datastore, is failing after circuit breaker timeout:  

      2021-11-16T03:18:22,156 | ERROR | opendaylight-cluster-data-akka.actor.default-dispatcher-15 | Shard                            | 206 - org.opendaylight.controller.repackaged-akka - 4.0.7 | Failed to persist event type [org.opendaylight.controller.cluster.raft.persisted.SimpleReplicatedLogEntry] with sequence number [222242] for persistenceId [member-1-shard-inventory-config].
      akka.pattern.CircuitBreaker$$anon$13: Circuit Breaker Timed out.
      2021-11-16T03:18:22,167 | INFO  | opendaylight-cluster-data-shard-dispatcher-33 | Shard                            | 211 - org.opendaylight.controller.sal-clustering-commons - 4.0.7 | Stopping Shard member-1-shard-inventory-config
      2021-11-16T03:18:22,169 | WARN  | opendaylight-cluster-data-akka.actor.default-dispatcher-14 | LocalThreePhaseCommitCohort      | 213 - org.opendaylight.controller.sal-distributed-datastore - 4.0.7 | Failed to prepare transaction member-1-datastore-config-fe-0-txn-111328-0 on backend
      java.lang.RuntimeException: Transaction aborted due to shutdown.
      	at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator.abortPendingTransactions(ShardCommitCoordinator.java:400) ~[bundleFile:?]
      	at org.opendaylight.controller.cluster.datastore.Shard.postStop(Shard.java:322) ~[bundleFile:?]
      	at akka.actor.Actor.aroundPostStop(Actor.scala:556) ~[bundleFile:?]
      	at akka.actor.Actor.aroundPostStop$(Actor.scala:556) ~[bundleFile:?]
      	at akka.persistence.AbstractPersistentActor.akka$persistence$Eventsourced$$super$aroundPostStop(PersistentActor.scala:295) ~[bundleFile:?]
      	at akka.persistence.Eventsourced.aroundPostStop(Eventsourced.scala:304) ~[bundleFile:?]
      	at akka.persistence.Eventsourced.aroundPostStop$(Eventsourced.scala:300) ~[bundleFile:?]
      	at akka.persistence.AbstractPersistentActor.aroundPostStop(PersistentActor.scala:295) ~[bundleFile:?]
      	at akka.actor.dungeon.FaultHandling.finishTerminate(FaultHandling.scala:240) ~[bundleFile:?]
      	at akka.actor.dungeon.FaultHandling.handleChildTerminated(FaultHandling.scala:329) ~[bundleFile:?]
      	at akka.actor.dungeon.FaultHandling.handleChildTerminated$(FaultHandling.scala:308) ~[bundleFile:?]
      	at akka.actor.ActorCell.handleChildTerminated(ActorCell.scala:411) ~[bundleFile:?]
      	at akka.actor.dungeon.DeathWatch.watchedActorTerminated(DeathWatch.scala:102) ~[bundleFile:?]
      	at akka.actor.dungeon.DeathWatch.watchedActorTerminated$(DeathWatch.scala:87) ~[bundleFile:?]
      	at akka.actor.ActorCell.watchedActorTerminated(ActorCell.scala:411) ~[bundleFile:?]
      	at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:513) ~[bundleFile:?]
      	at akka.actor.ActorCell.systemInvoke(ActorCell.scala:536) ~[bundleFile:?]
      	at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:295) ~[bundleFile:?]
      	at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:273) ~[bundleFile:?]
      	at akka.dispatch.Mailbox.run(Mailbox.scala:231) ~[bundleFile:?]
      	at akka.dispatch.Mailbox.exec(Mailbox.scala:243) ~[bundleFile:?]
      	at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290) [?:?]
      	at java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020) [?:?]
      	at java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656) [?:?]
      	at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594) [?:?]
      	at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183) [?:?]
      

      This happens to all 3 versions - Silicon, Phosphorus & Sulfur.

      Attachments

        # Subject Branch Project Status CR V

        Activity

          People

            rovarga Robert Varga
            sangwookha Sangwook Ha
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated: