Operating System: All
Platform: All

Cause Analysis:
The persisted data has been removed after the first shutdown of the node,at this point,when the node is restarted,each shard of the node is in the initialization state because the local persistence content is empty,that is,the logIndex of each shard is -1(indicating that the shard has not been synchronized to any data).
When the node is added to the cluster,the shard-leader on the other nodes synchronizes the data to the shard on the node through the raft protocol.Since shard-leader has done log compression, it is only possible to recover the data of the node via the snapshot message. At this point, the new startup node is closed, and it is possible that some of the shard's snapshot has not yet been completed.

Resolvent:
According to the above analysis, the status record of leader synchronization from snapshot to follower has no timeliness, which leads leader to judge snapshot synchronization with follower.The significance of this status record is to avoid sending snapshot messages repeatedly by leader.And the record is useless after the follower's shutdown,so the record cannot always be used as a basis for data synchronization between leader and follower.
To this end,we have added a timeliness check to this status record,the record is invalid in ten minutes(according to our business data scale, ten minutes is a very conservative time interval),and leader will continue to synchronize data to follower.

Our bug fix has passed 3 nodes test,we will push the code if someone can review.

Adjust severity to match last status in bugzilla.

Hi Guan,

do you have a link to the patch for review? This is a blocker bug for our nitrogen release we are hoping to release in a week.
The contributions are much appreciated.

Why is this all of a sudden a blocker for SR1? It wasn't a blocker for nitrogen plus this is an edge case that has been there all along (no regression).

I lowered the severity as I don't believe this needs to be a blocker.