https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/CONTROLLER-2044 controller <p>The cutover to Akka Artery and tell-based protocol (which puts more strain on persistence) crops up problems with serialization of our messages, like <a href="https://jira.opendaylight.org/browse/CONTROLLER-2037" title="Fail to serialize oversized message" class="issue-link" data-issue-key="CONTROLLER-2037"><del>CONTROLLER-2037</del></a> (estimates) and <a href="https://jira.opendaylight.org/browse/CONTROLLER-2043" title="Circuit breaker timeout with BGP and tell-based protocol" class="issue-link" data-issue-key="CONTROLLER-2043">CONTROLLER-2043</a> (raw volume).</p> <p>Given that Akka has deprecated Java serialization as the protocol to use and the mentioned issues, it would be prudent to implement a better serialization protocol around Payloads, both in order to remove dependency on Java serialization and to improve its efficiency.</p> <p>This mostly revolves around org.opendaylight.controller.cluster.raft.messages.Payload extensibility &#8211; we really want to attach a PayloadRegistry which will handle dispatch to serialization based on a single-byte type and base it on DataOutput/WritableObject interfaces. This will allow us to reduce the message overheads (which are significant).</p> <p>We should also deal with sal-akka-raft message serialization based on similar interface and tie it in with sal-akka-segmented-journal, so that it can operate more efficiently than through Serializable.</p> CONTROLLER-2044

Improve sal-akka-raft serialization protocol

Improvement High Confirmed Unresolved Unassigned Robert Varga pt Thu, 26 May 2022 20:19:44 +0000 Tue, 16 Jan 2024 09:23:06 +0000 10.0.0 9.0.1 clustering 0 3 <p>Even when we design and implement the new Serialization utils and the PayloadRegistry, we can't simply replace the old java serialization. Since this is a breaking change, we will need to support both the old solution as well as the new one. The old one will need to be deprecated first and removed in the subsequent major release.</p> <p>Right, except we are more flexible than that.</p> <p>Our upgrade matrix says 'upgrade from latest SR', which means we can very much evolve the format as long as we backport the bits needed for reading to active streams. See <a href="https://jira.opendaylight.org/browse/CONTROLLER-2056" title="Minimize serialization proxy names in controller.datastore.persisted" class="issue-link" data-issue-key="CONTROLLER-2056"><del>CONTROLLER-2056</del></a>, which will enable new proxies in Chlorine SR2 and Argon, with read-size of those bits being backported to Sulfur SR3 and Chlorine SR1.</p> <p><a href="https://jira.opendaylight.org/browse/CONTROLLER-2071" title="Switch to our fork of atomix-storage" class="issue-link" data-issue-key="CONTROLLER-2071"><del>CONTROLLER-2071</del></a> imports atomix-storage in a very minimal form.</p> <p><a href="https://jira.opendaylight.org/browse/CONTROLLER-2072" title="Remove Kryo from atomix-storage" class="issue-link" data-issue-key="CONTROLLER-2072">CONTROLLER-2072</a> further narrows the serialization interface down so as not to expose Kryo (and eliminate it from the picture while still retaining its serialization format).</p> <p>This issue needs to pick up from there. Next steps are to narrow down the interface between sal-akka-segmented-journal and sal-akka-raft.</p> <p><a href="https://jira.opendaylight.org/browse/CONTROLLER-2043" title="Circuit breaker timeout with BGP and tell-based protocol" class="issue-link" data-issue-key="CONTROLLER-2043">CONTROLLER-2043</a> has interplay here, as it provides data as to what our actual binary looks like. A brief analysis shows at least uuid being repeated, but certainly are other, deeper, connections to make.</p> <p>The end zone is <a href="https://jira.opendaylight.org/browse/CONTROLLER-2073" title="Ditch Akka persistence from sal-akka-raft" class="issue-link" data-issue-key="CONTROLLER-2073">CONTROLLER-2073</a>, which ties the entire stack together.</p> <p> </p> <p>The end-to-end serialization protocol needs to achieve something that  <a href="https://github.com/real-logic/aeron" class="external-link" target="_blank" rel="nofollow noopener">Aeron</a> has integrated: a single logical message (say, CommitTransactionPayload) is not written as a single byte[] prepared beforehand, but rather is scattered into multiple fragments which have some (possibly small) maximum size – in our world each such fragment would be a single SegmentedJournal entries. These then get presented to upper layers on reception, i.e. we are looking at doing <a href="#message-reassembly" target="_blank" rel="noopener">reassembly</a>.] In concrete terms this would mean that a ReplicatedLog entry is backed by multiple SegmentedJournal entries.</p> <p>For our implementation purposes, I think we should be okay by not exposing fragmentation explictly, but hiding it behind a DataOutput – which on writeInt() does the right thing behind the scenes. On reception, the callback would be a DataInput, which actually goes to multiple SegmentedJournal spans (suitably abstracted as ByteBuffer, or similar) and reads them as things are being read it.</p> <p>I am not sure this will cut it, though: the read side may be fine being synchronous, the write side, though, can hit storage allocation (i.e. when SegmentedJournal creates the next file) and at least in CSIT that operation can take almost a minute – and during that time we need to be able to handle RAFT messages, so as not to lose leadership (for example).</p> <p> <span class="image-wrap" style=""><a id="18710_thumb" href="https://jira.opendaylight.org/secure/attachment/18710/18710_Screenshot+from+2023-04-19+15-36-12.png" title="Screenshot from 2023-04-19 15-36-12.png" file-preview-type="image" file-preview-id="18710" file-preview-title="Screenshot from 2023-04-19 15-36-12.png"><img src="https://jira.opendaylight.org/secure/thumbnail/18710/_thumb_18710.png" style="border: 0px solid black" role="presentation"/></a></span> design of <a href="https://git.opendaylight.org/gerrit/c/controller/+/105485" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/c/controller/+/105485</a></p> <p>Right, and that's all centralized in one place and does not bring sal-akka-segmented-journal into the picture at all.</p> <p>So let's start at the bottom and design how will things get persisted in sal-akka-segmented-journal and atomix-storage, what interface will be available for sal-akka-raft to plug into. That includes how things will hit storage, but also how things will get restored &#8211; potentially in a different version.</p> <p>The goal is not to rely on Serializable anywhere in the stack.</p> <p>Alright, after spending a couple of days in the area, this is a tad more complicated.</p> <p>SegmentedJournal's API captures the RAFT journal, i.e. Journal&lt;E&gt; would really like to be Journal&lt;ReplicatedLogEntry&gt;, which is quite a bit different layer from where sal-akka-segmented-journal operates.</p> <p><a href="https://git.opendaylight.org/gerrit/c/controller/+/109487" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/c/controller/+/109487</a> shows a prototype outline of what we would like to see w.r.t. serialization API: each entry ends up writing multiple fragments and on read it is composed of a number of read only ByteBuffer (backed by the mapped file).</p> <p>The end play is <a href="https://jira.opendaylight.org/browse/CONTROLLER-2073" title="Ditch Akka persistence from sal-akka-raft" class="issue-link" data-issue-key="CONTROLLER-2073">CONTROLLER-2073</a>, which ends up integrating persistence into ReplicatedLog implementation, so that the implementation deals with snapshots, trimming, persistence etc. etc.</p> <p>Digging into this deeper, we have two technologies available off the shelf:<br/> 1. Aeron Archive, which uses multiple segments and allows querying it via replay, but the problem is this functionality assumes remote access and therefore there is no way to take a reference to the off-heap DirectBuffer for efficient fragment reassembly (i.e. for messages &gt;16MiB it is recommended to perform application-level reassembly). The up side is that Aeron has log replication freely available.<br/> 2. Chronicle Queue, which uses multiple segments and heavily assumes local control plus offers a very neat API to stored bytes (<a href="https://www.javadoc.io/static/net.openhft/chronicle-bytes/2.25ea3/net/openhft/chronicle/bytes/BytesMarshallable.html" class="external-link" target="_blank" rel="nofollow noopener">https://www.javadoc.io/static/net.openhft/chronicle-bytes/2.25ea3/net/openhft/chronicle/bytes/BytesMarshallable.html</a>) with possibly being able to retain references (BytesStore and ReferenceCounted interfaces). The down side is that log replication is a commercial feature.</p> <p>The BytesMarshallable interface certainly looks like something we'd like as the base model of how a payload gets serialized/deserialized.</p> <p>The second part of this is the fact that Payload users seem to want the associated Identifier &#8211; all relevant implementations have some sort of identifier and those which do not can fake it. Lifecycle is quite murky, but it would seem advantageous to move the identifier into an envelope, so it is available without deserializing the entire payload (as it is at the head of it!).</p> <p>That brings us to the Serdes API. As the mentioned patch shows, we want to side-step the need for writeObject() in sal-akka-segmented-journal by having a each Payload type:</p> <ul class="alternate" type="square"> <li>having a symbolic name (plain ASCII, i.e. 0-127, bytes, so encoding is single-byte)</li> <li>be dynamically bound to marshallers, so that each symbolic name has an assigned int32 (or smaller) unique to the set of known marshallers</li> </ul> <p>Based on this API, segmented-journal should be able to side-step writeObject() and thus eliminate the worst Java serialization overhead.</p> <p>In parallel to this, we need to fix the entry UUID duplications, which looks like something that is internal to the journal plugin and does not need any other help.</p> Blocks CONTROLLER-2073 CONTROLLER-2071 CONTROLLER-2072 Duplicate CONTROLLER-1969 Issue split CONTROLLER-2089 Relates CONTROLLER-2037 CONTROLLER-2043 Development Rank 0|i042h3: