https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/CONTROLLER-1632 controller <p>odl: distribution-karaf-0.6.0-20170411.050836-4837.zip</p> <p>The test scenario follows the steps of DOMDataBroker testing: Leader Isolation: partition heals within transaction timeout<br/> 1) start test-transaction producer, running at 1K tps, non-overlapping, from all nodes to a single shard<br/> 2) Isolate leader<br/> 3) Wait for followers to initiate election<br/> 4) Un-isolate leader<br/> 5) Wait for partition to heal - this was done within transaction timeout (30s)</p> <p>After the followers elected new leader, the node was rejoined (node unblocked via iptables).<br/> Then the test waited for all 3 nodes to be reported correctly. Few seconds after rejoin the node still reported IsolatedLeader raft state and then<br/> started to respond with http status 401 (unauthorized).<br/> This took</p> <p>Operating System: All<br/> Platform: All</p> CONTROLLER-1632

IOException in initiateCaptureSnapshot

Bug Verified Done Unassigned Peter Gubka Tue, 11 Apr 2017 10:00:26 +0000 Tue, 25 Jul 2023 08:24:29 +0000 Fri, 21 Apr 2017 12:37:10 +0000 clustering 0 2 <p>Attachment log.html has been added with description: robot log</p> <p>Attachment karaf_log_1.tar.gz has been added with description: karaf log</p> <p>Attachment karaf_log_2.tar.gz has been added with description: karaf log 2</p> <p>Attachment karaf_log_3.tar.gz has been added with description: karaf log 3</p> <p>Which shard are you looking at? In looking at the default-config shard, I see that node1 was the leader and was isolated:</p> <p>2017-04-11 08:47:33,069 | INFO | ult-dispatcher-4 | Shard | 226 - org.opendaylight.controller.sal-clustering-commons - 1.5.0.SNAPSHOT | member-1-shard-default-config (Leader) :- Switching from behavior Leader to IsolatedLeader, election term: 2</p> <p>Meanwhile node3 became leader on the other side of the partition with term 3:</p> <p>2017-04-11 08:47:28,644 | INFO | lt-dispatcher-42 | ShardManager | 233 - org.opendaylight.controller.sal-distributed-datastore - 1.5.0.SNAPSHOT | shard-manager-config: Received role changed for member-3-shard-default-config from PreLeader to Leader</p> <p>node1 switched to follower at:</p> <p>2017-04-11 08:47:38,854 | INFO | lt-dispatcher-29 | Shard | 226 - org.opendaylight.controller.sal-clustering-commons - 1.5.0.SNAPSHOT | member-1-shard-default-config (IsolatedLeader): Term 3 in "AppendEntriesReply <span class="error">&#91;term=3, success=false, followerId=member-2-shard-default-config, logLastIndex=20, logLastTerm=3, forceInstallSnapshot=false, payloadVersion=5, raftVersion=3&#93;</span>" message is greater than leader's term 2 - switching to Follower</p> <p>2017-04-11 08:47:38,855 | INFO | lt-dispatcher-29 | Shard | 226 - org.opendaylight.controller.sal-clustering-commons - 1.5.0.SNAPSHOT | member-1-shard-default-config (IsolatedLeader) :- Switching from behavior IsolatedLeader to Follower, election term: 3</p> <p>New leader node3 then installed a snapshot on node1 to catch it up:</p> <p>2017-04-11 08:47:40,693 | INFO | lt-dispatcher-34 | Shard | 226 - org.opendaylight.controller.sal-clustering-commons - 1.5.0.SNAPSHOT | member-3-shard-default-config (Leader): handleAppendEntriesReply: follower member-1-shard-default-config lastIndex 24 is ahead of our lastIndex 20 (snapshotIndex 17) - forcing install snaphot</p> <p>2017-04-11 08:47:40,758 | INFO | lt-dispatcher-83 | Shard | 226 - org.opendaylight.controller.sal-clustering-commons - 1.5.0.SNAPSHOT | member-1-shard-default-config: Applying snapshot on follower: Snapshot <span class="error">&#91;lastIndex=20, lastTerm=3, lastAppliedIndex=20, lastAppliedTerm=3, unAppliedEntries size=0, state=org.opendaylight.controller.cluster.datastore.persisted.ShardSnapshotState@51af049b, electionTerm=3, electionVotedFor=null, ServerConfigPayload=null&#93;</span></p> <p>I don't see a problem here.</p> <p>To fix my the problem description:<br/> after the rejoin the the test again iterates over all nodes and verifies the cluster status.<br/> Some time after the rejoin it reported the the of the node as IsolatedLeader. After some time Follower status should be reported. Instead of that the node started to respond with 401 (probably to anything). In the attached test case it responded 401 to GET for /restconf/modules.</p> <p>I changed the bug description too.</p> <p>(In reply to Peter Gubka from comment #5)<br/> &gt; To fix my the problem description:<br/> &gt; after the rejoin the the test again iterates over all nodes and verifies the<br/> &gt; cluster status.<br/> &gt; Some time after the rejoin it reported the the of the node as<br/> &gt; IsolatedLeader. After some time Follower status should be reported. Instead<br/> &gt; of that the node started to respond with 401 (probably to anything). In the<br/> &gt; attached test case it responded 401 to GET for /restconf/modules.<br/> &gt; <br/> &gt; I changed the bug description too.</p> <p>As I mentioned, from the logs the default config shard did transition from IsolatedLeader to Follower as expected. A 401 response means "Unauthorized" which means the credentials were rejected. This would not be an issue with clustering. You need to look at the robot logs to see what it's sending and to which node.</p> <p>Later on in the log for node3, I see this error repeated a few times:</p> <p>2017-04-11 08:47:56,268 | INFO | lt-dispatcher-33 | Shard | 226 - org.opendaylight.controller.sal-clustering-commons - 1.5.0.SNAPSHOT | member-3-shard-default-config (Leader): handleAppendEntriesReply: follower member-1-shard-default-config lastIndex 24 is ahead of our lastIndex 21 (snapshotIndex 17) - forcing install snaphot<br/> 2017-04-11 08:47:56,419 | WARN | lt-dispatcher-80 | ShardManager | 233 - org.opendaylight.controller.sal-distributed-datastore - 1.5.0.SNAPSHOT | Supervisor Strategy caught unexpected exception - resuming<br/> java.lang.RuntimeException: java.io.IOException: The # of bytes read from the imput stream, -1, does not match the expected # 3075<br/> at com.google.common.base.Throwables.propagate(Throwables.java:160)<span class="error">&#91;84:com.google.guava:18.0.0&#93;</span><br/> at org.opendaylight.controller.cluster.raft.behaviors.AbstractLeader.sendSnapshotChunk(AbstractLeader.java:831)<span class="error">&#91;227:org.opendaylight.controller.sal-akka-raft:1.5.0.SNAPSHOT&#93;</span><br/> at org.opendaylight.controller.cluster.raft.behaviors.AbstractLeader.initiateCaptureSnapshot(AbstractLeader.java:745)<span class="error">&#91;227:org.opendaylight.controller.sal-akka-raft:1.5.0.SNAPSHOT&#93;</span><br/> at org.opendaylight.controller.cluster.raft.behaviors.AbstractLeader.handleAppendEntriesReply(AbstractLeader.java:238)<span class="error">&#91;227:org.opendaylight.controller.sal-akka-raft:1.5.0.SNAPSHOT&#93;</span><br/> at org.opendaylight.controller.cluster.raft.behaviors.Leader.handleAppendEntriesReply(Leader.java:101)<span class="error">&#91;227:org.opendaylight.controller.sal-akka-raft:1.5.0.SNAPSHOT&#93;</span><br/> at org.opendaylight.controller.cluster.raft.behaviors.AbstractRaftActorBehavior.handleMessage(AbstractRaftActorBehavior.java:411)<span class="error">&#91;227:org.opendaylight.controller.sal-akka-raft:1.5.0.SNAPSHOT&#93;</span><br/> at org.opendaylight.controller.cluster.raft.behaviors.AbstractLeader.handleMessage(AbstractLeader.java:469)<span class="error">&#91;227:org.opendaylight.controller.sal-akka-raft:1.5.0.SNAPSHOT&#93;</span><br/> at org.opendaylight.controller.cluster.raft.behaviors.Leader.handleMessage(Leader.java:80)<span class="error">&#91;227:org.opendaylight.controller.sal-akka-raft:1.5.0.SNAPSHOT&#93;</span><br/> at org.opendaylight.controller.cluster.raft.RaftActor.possiblyHandleBehaviorMessage(RaftActor.java:322)<span class="error">&#91;227:org.opendaylight.controller.sal-akka-raft:1.5.0.SNAPSHOT&#93;</span><br/> at org.opendaylight.controller.cluster.raft.RaftActor.handleCommand(RaftActor.java:271)<span class="error">&#91;227:org.opendaylight.controller.sal-akka-raft:1.5.0.SNAPSHOT&#93;</span><br/> ...<br/> Caused by: java.io.IOException: The # of bytes read from the imput stream, -1, does not match the expected # 3075<br/> at org.opendaylight.controller.cluster.raft.behaviors.LeaderInstallSnapshotState.getNextChunk(LeaderInstallSnapshotState.java:129)<span class="error">&#91;227:org.opendaylight.controller.sal-akka-raft:1.5.0.SNAPSHOT&#93;</span><br/> at org.opendaylight.controller.cluster.raft.behaviors.AbstractLeader.sendSnapshotChunk(AbstractLeader.java:804)<span class="error">&#91;227:org.opendaylight.controller.sal-akka-raft:1.5.0.SNAPSHOT&#93;</span></p> <p>I think what happened is that when node1 was isolated, node3 tries to continuously send AppendEntries at the heartbeat interval so multiple messages are queued up and eventually get sent to node1 which results in multiple replies in quick succession where the follower's last index is higher and it tries to force a snapshot install. The first reply sends the snapshot chunk and drains the stream which causes the error when the second reply tries to send the same chunk. </p> <p>This error causes the shard actor to restart but I think this is orthogonal and wouldn't result in a 401 on the HTTP front-end, unless AAA was trying to access the credentials from the datastore but I don't think that info is stored there.</p> <p>I also noticed multiple messages like:</p> <p>2017-04-11 08:48:07,057 | INFO | lt-dispatcher-16 | Shard | 226 - org.opendaylight.controller.sal-clustering-commons - 1.5.0.SNAPSHOT | member-3-shard-default-config (Leader): follower member-1-shard-default-config last log term 2 conflicts with the leader's 3 - dec next index to -2</p> <p>I think this is benign. I pushed <a href="https://git.opendaylight.org/gerrit/#/c/54780" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/54780</a> to prevent decrementing next index below -1.</p> <p>It seems the 401 issue is more general and occurs with other tests as well. <a href="https://jira.opendaylight.org/browse/NETCONF-398" title="Carbon: odl respond with http status 401 in various suites" class="issue-link" data-issue-key="NETCONF-398"><del>NETCONF-398</del></a> was opened for that. I'll use this bug to address the IOException during install snapshot.</p> <p>Submitted <a href="https://git.opendaylight.org/gerrit/#/c/54898/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/54898/</a></p> Development External issue ID 8206 External issue URL Issue Type ODL SR Target Milestone Rank 0|i02rrr: