Uploaded image for project: 'controller'
  1. controller
  2. CONTROLLER-1534

Introduce 'dormant' Shard follower state

    XMLWordPrintable

Details

    • New Feature
    • Status: Confirmed
    • Resolution: Unresolved
    • None
    • None
    • clustering
    • None
    • Operating System: All
      Platform: All

    Description

      Our current implementation requires all followers to keep the Shard state in two places: the RaftActor journal and the DataTree. This means that the followers have to process incoming snapshots and payloads. This is an inherent trade-off.

      It gives us quicker failover times, as the follower DataTree is able to immediately start processing transactions. It also provides early safety net for catching failures, as payloads which fail to apply to the DataTree are immediately logged. It also provides the ability to serve DataTreeCandidates (although we do not implement it yet).

      On the other hand it costs us CPU time (to parse the snapshots and apply them) and heap (to hold the DataTree).

      Introduce an optional 'dormant' follower state, which will keep RaftActor state, but will not maintain the DataTree. This needs to be driven by the RaftActor, because the transition from dormant to active follower needs to happen before RaftActor acknowledges becoming the leader.

      This will not eliminate occasional spikes, as taking a journal snapshot will stil require a processing spike, where the DataTree is used. Proper resource sizing (journal size) and explicit ranged snapshotting can alleviate this by moving processing to off-peak hours.

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            Unassigned Unassigned
            rovarga Robert Varga
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated: