Two leaders for the same term emerged in a cluster which was restarted

This issue was seen on a system with a 5 node cluster. At some point the system was running out of disk space which was fixed and the nodes in the cluster were restarted. When the nodes were restarted two Leaders were seen for the default-config shard.

Here is some information collection from the system,

Current term on both nodes was : 22
Commit Index on all nodes seem to match the commit index on member-4
Commit Index on member-3 is less than the commit index on member-4

Committed Transaction Count on member-3 : 22
Committed Transaction Count on member-4: 32
Leadership Change Time on member-3 : 19:10:26.476
Leadership Change Time on member-4 : 19:10:26.463

Leadership Change count on both member-3/member-4 is 1. This seems to indicate that they both requested to become leader on startup and both got enough votes to become Leader.

Member Name Last Voted For
member-1 member-3
member-2 member-3
member-3 member-3
member-4 member-4
member-5 member-4

Missing info

• Debugging logging could have shown the following information,
• Who voted to whom when
• Who changed their vote in term 22