[CONTROLLER-2026] Circuit breaker timeout

Relates — Tue, 2 May 2023 18:48:44 +0000
OpenFlow Plugin longevity test, which generates a lot of updates in the inventory datastore, is failing after circuit breaker timeout:
2021-11-16T03:18:22,156 | ERROR | opendaylight-cluster-data-akka.actor.default-dispatcher-15 | Shard                            | 206 - org.opendaylight.controller.repackaged-akka - 4.0.7 | Failed to persist event type [org.opendaylight.controller.cluster.raft.persisted.SimpleReplicatedLogEntry] with sequence number [222242] for persistenceId [member-1-shard-inventory-config].
akka.pattern.CircuitBreaker$$anon$13: Circuit Breaker Timed out.
2021-11-16T03:18:22,167 | INFO  | opendaylight-cluster-data-shard-dispatcher-33 | Shard                            | 211 - org.opendaylight.controller.sal-clustering-commons - 4.0.7 | Stopping Shard member-1-shard-inventory-config
2021-11-16T03:18:22,169 | WARN  | opendaylight-cluster-data-akka.actor.default-dispatcher-14 | LocalThreePhaseCommitCohort      | 213 - org.opendaylight.controller.sal-distributed-datastore - 4.0.7 | Failed to prepare transaction member-1-datastore-config-fe-0-txn-111328-0 on backend
java.lang.RuntimeException: Transaction aborted due to shutdown.
	at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator.abortPendingTransactions(ShardCommitCoordinator.java:400) ~[bundleFile:?]
	at org.opendaylight.controller.cluster.datastore.Shard.postStop(Shard.java:322) ~[bundleFile:?]
	at akka.actor.Actor.aroundPostStop(Actor.scala:556) ~[bundleFile:?]
	at akka.actor.Actor.aroundPostStop$(Actor.scala:556) ~[bundleFile:?]
	at akka.persistence.AbstractPersistentActor.akka$persistence$Eventsourced$$super$aroundPostStop(PersistentActor.scala:295) ~[bundleFile:?]
	at akka.persistence.Eventsourced.aroundPostStop(Eventsourced.scala:304) ~[bundleFile:?]
	at akka.persistence.Eventsourced.aroundPostStop$(Eventsourced.scala:300) ~[bundleFile:?]
	at akka.persistence.AbstractPersistentActor.aroundPostStop(PersistentActor.scala:295) ~[bundleFile:?]
	at akka.actor.dungeon.FaultHandling.finishTerminate(FaultHandling.scala:240) ~[bundleFile:?]
	at akka.actor.dungeon.FaultHandling.handleChildTerminated(FaultHandling.scala:329) ~[bundleFile:?]
	at akka.actor.dungeon.FaultHandling.handleChildTerminated$(FaultHandling.scala:308) ~[bundleFile:?]
	at akka.actor.ActorCell.handleChildTerminated(ActorCell.scala:411) ~[bundleFile:?]
	at akka.actor.dungeon.DeathWatch.watchedActorTerminated(DeathWatch.scala:102) ~[bundleFile:?]
	at akka.actor.dungeon.DeathWatch.watchedActorTerminated$(DeathWatch.scala:87) ~[bundleFile:?]
	at akka.actor.ActorCell.watchedActorTerminated(ActorCell.scala:411) ~[bundleFile:?]
	at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:513) ~[bundleFile:?]
	at akka.actor.ActorCell.systemInvoke(ActorCell.scala:536) ~[bundleFile:?]
	at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:295) ~[bundleFile:?]
	at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:273) ~[bundleFile:?]
	at akka.dispatch.Mailbox.run(Mailbox.scala:231) ~[bundleFile:?]
	at akka.dispatch.Mailbox.exec(Mailbox.scala:243) ~[bundleFile:?]
	at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290) [?:?]
	at java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020) [?:?]
	at java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656) [?:?]
	at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594) [?:?]
	at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183) [?:?]
This happens to all 3 versions - Silicon, Phosphorus & Sulfur.
OpenDaylight JIRA

[CONTROLLER-2026] Circuit breaker timeout
