Operating System: All
Platform: All

> See failure in csit [0] when journal and snapshots directories are deleted before re-start.

Unfortunately, Sandbox run [2] shows the suite fails also without deleting the directories. There may be an unrelated bug and this is just a temporary side effect.

[2] https://jenkins.opendaylight.org/sandbox/job/controller-csit-3node-clustering-all-carbon/1

The only way to fix this is to resync client generations across all member nodes when we bring the frontend up. That is extremely problematic, given that not all nodes may actually be present. While this is worth considering, it certainly lacks any sort of priority.

On the side of integration: do not wipe the entire persistence, just the backend store bits.

I finally looked at Raft algorithm. It really only works if persistence is reliable. If a member loses its persisted data on restart, it has to be treated as a brand new cluster member, that includes all the operations for cluster reconfiguration (remove the old incarnation from raft, add the new incarnation).

As the usual "member-2" identifier does not change with new incarnation, there should be some random nonce generated (or restored from persisted data) when a member starts. I feel "how to make raft work when persisted data can be lost" is a problem somebody already solved, it is just hard to google for that.

The key thing to realize here is that you are using 'persistence' as a black box.

There are two separate sets of data in akka persistence:

• Shard journal, which is replicated using RAFT
• Frontend generation records, which is not replicated

You cannot assume that you can wipe the entire persistence store, as that will include non-replicated private data.

If you wipe the shard journal only, things should recover.

(In reply to Vratko Polák from comment #3)
> I finally looked at Raft algorithm. It really only works if persistence is
> reliable. If a member loses its persisted data on restart, it has to be
> treated as a brand new cluster member, that includes all the operations for
> cluster reconfiguration (remove the old incarnation from raft, add the new
> incarnation).
>
> As the usual "member-2" identifier does not change with new incarnation,
> there should be some random nonce generated (or restored from persisted
> data) when a member starts. I feel "how to make raft work when persisted
> data can be lost" is a problem somebody already solved, it is just hard to
> google for that.

Raft works fine if a member is restarted with its persisted data cleaned. On restart the leader will install a snapshot to sync it up and restore its state to match the leader's state.

(In reply to Robert Varga from comment #4)
> The key thing to realize here is that you are using 'persistence' as a black
> box.
>
> There are two separate sets of data in akka persistence:
> - Shard journal, which is replicated using RAFT
> - Frontend generation records, which is not replicated
>
> You cannot assume that you can wipe the entire persistence store, as that
> will include non-replicated private data.
>
> If you wipe the shard journal only, things should recover.

If you wipe the shard journal, you also have to wipe any snapshots for those shards.

I'm not entirely clear what the test does but it seems the problem is that a member is restarted with a clean persistence store and thus restarts with a generation ID of 0 but the remote shard still has a cached front-end state with generation 2. Since the new generation ID is less it rejects it as retired/stale.

Is it necessary for the test to wipe the persistence store? If so perhaps it could retain just the front-end generation state, at least for now so it doesn't fail.

Longer term I think we need to revisit the retired generation logic. Does it need to fail if the new generation ID < the cached generation? At what point does the shard purge the cached front-end state? This scenario is valid and will occur in production where a node is restarted clean for some reason. This will also occur when a node is restored from a backup. For the latter, perhaps the front-end generation records should be included in the backup.

> do not wipe the entire persistence

By the way, we stopped wiping at all now [3], so this does not affect existing CSIT results anymore.

[3] https://git.opendaylight.org/gerrit/54264

> revisit the retired generation logic

Is the generation number just a (part of) identifier of a session between client and backend? The the client should not be the only party deciding its value.

I envision this:
A client incarnation sends "open" message (with its current counter 0). Backend replies "ok, use generation 1", and accepts requests with generation 1.
Now client crashes, loses everything. Another client incarnation sends "open" with counter 0. Backend replies "ok, use generation 2" and starts ignoring requests with generation 1.
The point is, the backend is the side choosing generation numbers (based on client lower bound), and it never replies with the same number twice. Perhaps the backend even persists the new value to Raft log, so future backend incarnations never start listening to outdated clients.

I agree with Vratko that the generation-ID (if it's required for correct functioning of clustering) really needs to be something that can be learned by clients and is consistently stored in the datastore so that it doesn't need to be somehow magically recreated and can't accidentally get into a value which is out of range and hang everything.

No matter who generates/stores the generation Id (FE or BE), it still needs to be persisted and will suffer the same problem if the persistent store is wiped clean.

The reason for the generation Id in the first place was to alleviate the issue where a read tx was created with say id 10 on a remote leader, then the node that created the tx was restarted. If it cycled fast enough, which tends to happen in robot tests, it could generate id 10 again before the remote leader had timed out and removed it. Hence the generation Id is generated and persisted by the FE.

I think we should just remove the retired generation Id checking altogether in the shard.

> No matter who generates/stores the generation Id (FE or BE), it still needs
> to be persisted and will suffer the same problem if the persistent store is
> wiped clean.

Backend becomes active only on the shard leader, and to become the leader majority has to agree the reported state is updated enough.
I do not know what are the conditions for fronted to be active, but I believe "follower out of sync without knowing that" has active frontend (not sure about "follower just started").

A reply from backend may inform the frontend to wait for sync. That would help if everything important is in the raft committed state (as opposed to just in locally persisted store).

> I think we should just remove the retired generation Id checking altogether
> in the shard.

That would be best (if possible).

No we should not, it detects when a FE inconsistency is detected. If you wipe frontend state, that is a manual and exceptional transition, which requires a manual recovery – it must be ensured that the newer FE generation will never (ever) come around.

The proper way of addressing this is to add an operator option for the newly-created FE state to start counting not at 0, but at some operator-specified number.

So if the journal/snapshots is cleaned for DR, the end-user operator has to perform a manual operation to alleviate this issue? IMO, that is not acceptable for those that have to document/explain and support it - currently this scenario is seamless and should remain that way. skitt vorburger what do you guys think from the RH downstream perspective?

I think the removal of journal/snapshots needs to be documented to differentiated between frontend and backend state – if differentiating in documentation is not enough, we need to reconfigure akka pesistence to use two separate directories.

I agree with tpantelis; we run into this issue fairly regularly (thanks for the ping, I’m investigating on a bug just like this...).

Incidentally, what is the manual operation? Does it exist?

It does not currently exist – we are still discussion how to fix it. There are three options:

1. different directories for FE and BE state
2. modify DR guidance to not wipe entire directories
3. implement and document the manual generation bump

So the only scenario that I know of where this occurs is the case where the journal/snapshots is deleted where the FE counter is reset to 0. In this case it's OK to ignore the inconsistency and clear the stale state. rovarga what other real production scenario(s) do you see where this could occur?

The counter guards against the old generation being left behind – for example a failure of the FE to die completely, such that two generations end up running, or its messages still being somewhere in flight (akka, requeues, whatever) and arriving after the new generation has established contact and thereby becoming operational. Granted, it is largely hypothetical, but if such a bug ever happens, we could very easily end up with corrupted data and diagnosing the root cause would be extremely problematic.

Wiping frontend state shows up exactly this sort of bug to the backend, and hence the guard kicks in. The solution is either adjust FE so it is a newer than anything observed by backend, or purging all knowledge of all previous generations from BE. I think the former is a lot easier.

>> 3. implement and document the manual generation bump

> adjust FE so it is a newer than anything observed by backend

Why does it have to be manual? On startup, the frontend could ask the backed for the latest observed generation and use that+1 as its generation.

yeah - I think if the backend sees that new FE counter is 0 then assume it was restarted clean and purge the cached state and proceed.

That would mean that startup needs to be blocked until we have connectivity to all backend shards, they are stable and can give us an authoritative information. That communication is also subject to round-trip delays, hence by the time we get that information it is already stale.

The authoritative source of its current generation is the frontend – it's eight bytes of information, which simply needs to be restored without relying on the normal communication clusters.

That won't work, as if ever there are to FEs with counter=0, you will mistake them for the same entity, which will break like hell. Furthermore that various shards will observe the resurrected FE at different times, possibly with the FE restarting in between – at which point when BE sees the resurrected FE, counter != 0 --> boom.

Guys, I have thought about this long and hard, and a locally-maintained monotonic counter is really the only safe option.

Then perhaps let the BE throw the ex and have the FE realize that it was just restarted, sync to the BE generation # and persist, then re-issue the request. Let's discuss this next Tue. As it stands now, this introduces a failure scenario with tell-based that wasn't present before and requiring some manual operator intervention to resolve it is not acceptable/viable in a real production environment.

The simplest way I can see out of this is to instruct shards to forget about the FE's existence – i.e. have them remove the client from FrontendMetadata and persist a record of that purge.

Going off for the year, putting this back into backlog.

> Add an option to allow CDS FE to start its generation counting from non-zero
> Status: Resolved

Ok. We are missing a launcher script that would execute the call from 85373 and use the response to set the correct value for 85397.
But in my initial description I was not demanding a fully automated solution, so this ticket can be considered closed indeed.
I believe Int/Test people will open another ticket if such a launcher script is needed (and if they want to test such "persistence lost" scenarios).