https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/CONTROLLER-1512 controller <p>RaftActor implements time-sensitive messaging exchange, but it is also subclassed to provide subclass operations.</p> <p>When the subclass is targeted with a lot of messages, this can lead to timeouts in the RAFT underlay &#8211; hence potentially disrupting stability.</p> <p>Akka provides a way to divide classes into two:</p> <ul class="alternate" type="square"> <li>make important messages implement akka.dispatch.ControlMessage</li> <li>use one of ControlAwareMailboxes</li> </ul> <p>ControlAwareMailboxes use two internal queues, giving priority to delivering messages which are instanceof ControlMessage and delivering other messages only when no ControlMessage is present.</p> <p>This way we can ensure that the RAFT algorithm operates isolated from any floods incurred by the implementation above, leading to better stability.</p> <p>Performance impact and interactions with persistence need to be evaluated before this becomes our default.</p> <p>Operating System: All<br/> Platform: All</p> CONTROLLER-1512

sal-akka-raft: Use akka.dispatch.ControlMessage for messages

Bug Resolved Done Tom Pantelis Robert Varga Mon, 18 Apr 2016 15:13:52 +0000 Tue, 25 Jul 2023 08:24:09 +0000 Mon, 5 Jun 2017 14:59:39 +0000 clustering 0 4 <p>HI Robert, </p> <p>This implies that the order would be altered when compared to current order of delivery to Shard. </p> <p>Would there be any changes in functional behavior due to this alteration of order? For instance, transaction-related messages (non-RAFT) of shard get reordered with replication related (RAFT) messages </p> <p>Regards<br/> Muthu</p> <p>The ordering does not really matter, as the messages are inherently coming from two unrelated sources: applications and RAFT peers.</p> <p>As discussed on this week's call, it actually makes sense to make RaftActor a completely separate actor, with Shard being just talking to it using messages.</p> <p>There is not an issue for that, but if Ashutosh does not open one by Friday, I will open it myself (and make this issue blocked by that one).</p> <p>Even when we separate the two actors, the interchange is logically split into two classes:</p> <p>RaftActor &lt;-&gt; RaftActor: ControlMessages, since RAFT convergence is the top priority:</p> <p>Shard -&gt; RaftActor: normal message, just a request to persist a Payload</p> <p>RaftActor -&gt; Shard: ControlMessages, as these propagate role (Leader/Follower) and we do not want to reorder w.r.t. responses to Payload persistence</p> <ul> <li>-&gt; Shard: normal messages, as these are work items which generate code and therefore need to be subject to backpressure when RAFT is in flux and cannot keep up. Most notably this will favor delivering Payload response messages before client requests, because without them the system is not making forward progres.</li> </ul> <p>Currently we use the SingleConsumerOnlyUnboundedMailbox for shards which is a "high-performance, multiple producer—single consumer, unbounded MailboxType". But it also says "Currently this queue is slower for some benchmarks than the ConcurrentLinkedQueue from JDK 8 that is used by default, so be sure to measure the performance in your particular setting in order to determine which one to use."</p> <p>To switch to a ControlAwareMailbox, it looks like we'd go with the UnboundedControlAwareMailbox (I don't see any other option other than BoundedControlAwareMailbox). Looking at the scala code for UnboundedControlAwareMailbox.MessageQueue:</p> <p>object UnboundedControlAwareMailbox {<br/> class MessageQueue extends UnboundedControlAwareMessageQueueSemantics with java.io.Serializable </p> { val controlQueue: Queue[Envelope] = new ConcurrentLinkedQueue[Envelope]() val queue: Queue[Envelope] = new ConcurrentLinkedQueue[Envelope]() } <p>}</p> <p>it uses ConcurrentLinkedQueue for both queues. There's slight overhead to check if the Envelope is a ControlMessage but that looks negligible. </p> <p>We can add the configuration with a knob in the akka.conf to select which mailbox to use for shards.</p> <p>Patches: <a href="https://git.opendaylight.org/gerrit/#/q/status:open+project:controller+branch:master+topic:bug/5740" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/q/status:open+project:controller+branch:master+topic:bug/5740</a></p> <p>This should help <a href="https://jira.opendaylight.org/browse/CONTROLLER-1675" title="Leadership transfer failed: Follower is not ready to become leader" class="issue-link" data-issue-key="CONTROLLER-1675">CONTROLLER-1675</a> too.</p> Blocks CONTROLLER-1513 CONTROLLER-1675 Development External issue ID 5740 External issue URL Issue Type ODL SR Target Milestone Rank 0|i02r13: