https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/CONTROLLER-1734 controller <p>The robot symptom is basically the same as in <a href="https://jira.opendaylight.org/browse/CONTROLLER-1723" title="RequestTimeoutException after ~250s after removing module shard replica" class="issue-link" data-issue-key="CONTROLLER-1723"><del>CONTROLLER-1723</del></a> but the test scenario is different, so this is perhaps a different bug, especially when the suite when <a href="https://jira.opendaylight.org/browse/CONTROLLER-1723" title="RequestTimeoutException after ~250s after removing module shard replica" class="issue-link" data-issue-key="CONTROLLER-1723"><del>CONTROLLER-1723</del></a> appeared first is now passing.</p> <p>This suite starts transaction writer (duration 180 seconds) on each member, then isolates the leader for 40 seconds, the rejoins it again. It is expected each writer finishes successfully. Two days ago two writers were passing, one failed on <a href="https://jira.opendaylight.org/browse/CONTROLLER-1708" title="write-transactions does not respond (no evident reason)" class="issue-link" data-issue-key="CONTROLLER-1708"><del>CONTROLLER-1708</del></a>. Yesterday (two releng runs) this bug appeared alongside <a href="https://jira.opendaylight.org/browse/CONTROLLER-1708" title="write-transactions does not respond (no evident reason)" class="issue-link" data-issue-key="CONTROLLER-1708"><del>CONTROLLER-1708</del></a>, and today only this symptom is there <span class="error">&#91;0&#93;</span>. Two writers failed (161s ans 181s reported), one passed (the one on the new leader), not sure how reliable that is.</p> <p>Sandbox run with verbose log also sees <span class="error">&#91;1&#93;</span> this bug.</p> <p><span class="error">&#91;0&#93;</span> <a href="https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/767/log.html.gz#s1-s28-t1-k2-k25-k1-k1" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/767/log.html.gz#s1-s28-t1-k2-k25-k1-k1</a><br/> <span class="error">&#91;1&#93;</span> <a href="https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-li-only-carbon/17/log.html.gz#s1-s2-t1-k2-k25-k1-k1" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-li-only-carbon/17/log.html.gz#s1-s2-t1-k2-k25-k1-k1</a></p> <p>Operating System: All<br/> Platform: All</p> CONTROLLER-1734

RequestTimeoutException after ~250s after brief isolation

Bug Resolved Done Unassigned Vratko Polak Fri, 30 Jun 2017 12:44:48 +0000 Tue, 25 Jul 2023 08:24:44 +0000 Mon, 18 Sep 2017 12:32:04 +0000 clustering 0 2 <p>Recent Sandbox occurrence: <span class="error">&#91;2&#93;</span>.</p> <p><span class="error">&#91;2&#93;</span> <a href="https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-li-only-carbon/1/log.html.gz#s1-s2-t1-k2-k25-k1-k1" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-li-only-carbon/1/log.html.gz#s1-s2-t1-k2-k25-k1-k1</a></p> <p>Contrary to most other bugs, this show different symptom with <span class="error">&#91;3&#93;</span> codebase. Instead of RequestTimeoutException, writer responds <span class="error">&#91;4&#93;</span> with "Backend timeout in state READY":</p> <p>Caused by: java.util.concurrent.TimeoutException: Backend timeout in state READY after 15000ms\n\tat org.opendaylight.controller.cluster.datastore.ShardDataTree.checkForExpiredTransactions(ShardDataTree.java:999)\n\tat org.opendaylight.controller.cluster.datastore.Shard.handleNonRaftCommand(Shard.java:333)\n\tat org.opendaylight.controller.cluster.raft.RaftActor.handleCommand(RaftActor.java:270)\n\tat org.opendaylight.controller.cluster.common.actor.AbstractUntypedPersistentActor.onReceiveCommand(AbstractUntypedPersistentActor.java:31)\n\tat akka.persistence.UntypedPersistentActor.onReceive(PersistentActor.scala:170)\n\tat org.opendaylight.controller.cluster.common.actor.MeteringBehavior.apply(MeteringBehavior.java:104)\n\tat akka.actor.ActorCell$$anonfun$become$1.applyOrElse(ActorCell.scala:544)\n\t... 14 more</p> <p><span class="error">&#91;3&#93;</span> <a href="https://git.opendaylight.org/gerrit/#/c/60032/2" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/60032/2</a><br/> <span class="error">&#91;4&#93;</span> <a href="https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-li-only-carbon/3/log.html.gz#s1-s2-t1-k2-k25-k1-k1" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-li-only-carbon/3/log.html.gz#s1-s2-t1-k2-k25-k1-k1</a></p> <p>Code <span class="error">&#91;5&#93;</span> does NOT fix <span class="error">&#91;6&#93;</span> this.</p> <p><span class="error">&#91;5&#93;</span> <a href="https://git.opendaylight.org/gerrit/#/c/60033/5" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/60033/5</a><br/> <span class="error">&#91;6&#93;</span> <a href="https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-li-only-carbon/4/log.html.gz#s1-s2-t1-k2-k25-k1-k1" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-li-only-carbon/4/log.html.gz#s1-s2-t1-k2-k25-k1-k1</a></p> <p>&gt; Code <span class="error">&#91;5&#93;</span> does NOT fix <span class="error">&#91;6&#93;</span> this.</p> <p>This weeks run in the same <span class="error">&#91;5&#93;</span> code shows it can cause one writer to change its message.<br/> In this test, each member has a writer, and a leader (member-3) is isolated for a short time needed for the new leader (member-2) to get elected, after which member-3 is rejoined back. Member-2 writer finishes correctly and member-3 writer shows the original symptom of RequestTimeoutException.</p> <p>But member-1 writer returns <span class="error">&#91;7&#93;</span> the new "Backend timeout in state READY" message. This might help with other bugs predominantly affecting the member which starys follower.</p> <p><span class="error">&#91;7&#93;</span> <a href="https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-li-only-carbon/1/log.html.gz#s1-s2-t1-k2-k25-k1-k2-k1-k1-k1" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-li-only-carbon/1/log.html.gz#s1-s2-t1-k2-k25-k1-k2-k1-k1-k1</a></p> <p>Today, releng showed a different <span class="error">&#91;8&#93;</span> writer response:<br/> {"errors":{"error":[{"error-type":"application","error-tag":"operation-failed","error-message":"Final submit was timed out by the test provider or was interrupted","error-info":"java.util.concurrent.TimeoutException: Collection did not finish in 300 seconds\n\tat org.opendaylight.controller.clustering.it.provider.impl.AbstractTransactionHandler.checkComplete(AbstractTransactionHandler.java:187)\n\tat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:266)\n</p> <p>But looking at karaf.log <span class="error">&#91;9&#93;</span> I believe that is what this bug looks like when "~250s" is longer than 300s.</p> <p><span class="error">&#91;8&#93;</span> <a href="https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/778/log.html.gz#s1-s36-t3-k2-k13-k3-k1-k1" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/778/log.html.gz#s1-s36-t3-k2-k13-k3-k1-k1</a><br/> <span class="error">&#91;9&#93;</span> <a href="https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/778/odl1_karaf.log.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/778/odl1_karaf.log.gz</a></p> <p>One run on Sandbox with code <span class="error">&#91;10&#93;</span> had the usually failing test pass <span class="error">&#91;11&#93;</span>. Will retry to see it that is reliable.</p> <p><span class="error">&#91;10&#93;</span> <a href="https://git.opendaylight.org/gerrit/#/c/60137/4" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/60137/4</a><br/> <span class="error">&#91;11&#93;</span> <a href="https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-li-only-carbon/5/log.html.gz#s1-s2-t1-k2-k25-k1-k1" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-li-only-carbon/5/log.html.gz#s1-s2-t1-k2-k25-k1-k1</a></p> <p>Is this still happening?</p> <p>&gt; Is this still happening?</p> <p>This is no longer happening, marking as fixed.</p> Development External issue ID 8782 External issue URL Issue Type Rank 0|i02sef: