Uploaded image for project: 'controller'
  1. controller
  2. CONTROLLER-1675

Leadership transfer failed: Follower is not ready to become leader

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Confirmed
    • Resolution: Unresolved
    • Affects Version/s: unspecified
    • Fix Version/s: None
    • Component/s: clustering
    • Labels:
      None
    • Environment:

      Operating System: All
      Platform: All

    • External issue ID:
      8446

      Description

      This happened [0] in CSIT for module-based shard with tell-based protocol. Karaf.log [1] shows large number of warnings such as:

      2017-05-14 06:18:34,666 | WARN | lt-dispatcher-22 | FrontendClientMetadataBuilder | 199 - org.opendaylight.controller.sal-distributed-datastore - 1.5.0.Carbon | member-1-shard-default-config: Unknown history for purged transaction member-3-datastore-config-fe-0-chn-2-txn-1522-0, ignoring
      

      Here is the error and five infos before it:

       2017-05-14 06:18:34,674 | INFO | lt-dispatcher-28 | ShardManager | 199 - org.opendaylight.controller.sal-distributed-datastore - 1.5.0.Carbon | shard-manager-config Received follower initial sync status for member-1-shard-default-config status sync done false
       2017-05-14 06:18:34,716 | INFO | ult-dispatcher-2 | ShardManager | 199 - org.opendaylight.controller.sal-distributed-datastore - 1.5.0.Carbon | shard-manager-config Received follower initial sync status for member-1-shard-default-config status sync done true
       2017-05-14 06:18:34,716 | INFO | ult-dispatcher-2 | ShardManager | 199 - org.opendaylight.controller.sal-distributed-datastore - 1.5.0.Carbon | shard-manager-config Received follower initial sync status for member-1-shard-default-config status sync done false
       2017-05-14 06:18:34,724 | INFO | lt-dispatcher-29 | ShardManager | 199 - org.opendaylight.controller.sal-distributed-datastore - 1.5.0.Carbon | shard-manager-config Received follower initial sync status for member-1-shard-default-config status sync done true
       2017-05-14 06:18:34,724 | INFO | lt-dispatcher-29 | ShardManager | 199 - org.opendaylight.controller.sal-distributed-datastore - 1.5.0.Carbon | shard-manager-config: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-1-shard-default-config, leaderId=null, leaderPayloadVersion=5]
       2017-05-14 06:18:44,695 | ERROR | lt-dispatcher-21 | ClusterAdminRpcService | 201 - org.opendaylight.controller.sal-cluster-admin-impl - 1.5.0.Carbon | Leadership transfer failed for shard default.
       org.opendaylight.controller.cluster.raft.LeadershipTransferFailedException: Failed to transfer leadership to member-1-shard-default-config. Follower is not ready to become leader
       at org.opendaylight.controller.cluster.raft.RaftActor$1.onFailure(RaftActor.java:304)
       at org.opendaylight.controller.cluster.raft.RaftActorLeadershipTransferCohort.finish(RaftActorLeadershipTransferCohort.java:187)
       at org.opendaylight.controller.cluster.raft.RaftActorLeadershipTransferCohort.abortTransfer(RaftActorLeadershipTransferCohort.java:138)
       at org.opendaylight.controller.cluster.raft.RaftActorLeadershipTransferCohort$1.doCancel(RaftActorLeadershipTransferCohort.java:112)
       at org.opendaylight.controller.cluster.raft.TimedRunnable.cancel(TimedRunnable.java:47)
       at org.opendaylight.controller.cluster.raft.TimedRunnable.lambda$new$0(TimedRunnable.java:33)
       at org.opendaylight.controller.cluster.raft.RaftActor.handleCommand(RaftActor.java:264)
       at org.opendaylight.controller.cluster.common.actor.AbstractUntypedPersistentActor.onReceiveCommand(AbstractUntypedPersistentActor.java:31)
       at akka.persistence.UntypedPersistentActor.onReceive(PersistentActor.scala:170)
       at org.opendaylight.controller.cluster.common.actor.MeteringBehavior.apply(MeteringBehavior.java:104)
       at akka.actor.ActorCell$$anonfun$become$1.applyOrElse(ActorCell.scala:544)
       at akka.actor.Actor$class.aroundReceive(Actor.scala:497)
       at akka.persistence.UntypedPersistentActor.akka$persistence$Eventsourced$$super$aroundReceive(PersistentActor.scala:168)
       at akka.persistence.Eventsourced$$anon$1.stateReceive(Eventsourced.scala:664)
       at akka.persistence.Eventsourced$class.aroundReceive(Eventsourced.scala:183)
       at akka.persistence.UntypedPersistentActor.aroundReceive(PersistentActor.scala:168)
       at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
       at akka.actor.ActorCell.invoke(ActorCell.scala:495)
       at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
       at akka.dispatch.Mailbox.run(Mailbox.scala:224)
       at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
       at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
       at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
       at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
       at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
      

       

      [0] https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/684/archives/log.html.gz#s1-s29-t3-k2-k5-k3-k1-k4-k7-k1
      [1] https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/684/archives/odl1_karaf.log.gz

        Attachments

          Issue Links

          # Subject Branch Project Status CR V

            Activity

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              vrpolak Vratko Polak
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Dates

                Created:
                Updated: