https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/NETCONF-359 netconf <p>this is affected some CSIT jobs and makes them spend a lot of wasted time.</p> <p>This email thread sort of explains it:</p> <p><a href="https://lists.opendaylight.org/pipermail/integration-dev/2017-January/008967.html" class="external-link" target="_blank" rel="nofollow noopener">https://lists.opendaylight.org/pipermail/integration-dev/2017-January/008967.html</a></p> <p>to reproduce you can bring up a three node cluster. then kill the leader<br/> node and one of the followers. the remaining node that stays up will be in<br/> Candidate state. next send a simple rest request (e.g. /restconf/streams )<br/> to this node. The request will be open and pending for 20s before it's<br/> answered.</p> <p>in csit where there tend to be a lot of these requests, when this <br/> is seen it can severely add to the runtime of the test.</p> <p>If possible, even if the response needs to be an error, the response should<br/> come back right away. It seems there is some internal timer/timeout <br/> getting in the way here, since it's always 20s.</p> <p>Operating System: All<br/> Platform: All</p> NETCONF-359

restconf response taking 20s when sent to isolated Candidate node.

Bug Verified Done Jamo Luhrsen Jamo Luhrsen Fri, 3 Mar 2017 06:12:40 +0000 Fri, 15 Mar 2019 22:22:36 +0000 Fri, 31 Aug 2018 18:29:19 +0000 restconf-nb 0 5 <p>This is by design. It tries to fulfill the request by waiting for a new leader to be elected for a period of time (2 X the election timeout, which is 10 sec by default).</p> <p>This is the way it currently works however Robert is changing the front-end implementation with the <a href="https://jira.opendaylight.org/browse/CONTROLLER-1483" title="akka.pattern.AskTimeoutException on follower while BGP peer introduces 1M prefixes" class="issue-link" data-issue-key="CONTROLLER-1483"><del>CONTROLLER-1483</del></a> patch series so not sure what the exact behavior will be.</p> <p>(In reply to Tom Pantelis from comment #1)<br/> &gt; This is by design. It tries to fulfill the request by waiting for a new<br/> &gt; leader to be elected for a period of time (2 X the election timeout, which<br/> &gt; is 10 sec by default).<br/> &gt; <br/> &gt; This is the way it currently works however Robert is changing the front-end<br/> &gt; implementation with the <a href="https://jira.opendaylight.org/browse/CONTROLLER-1483" title="akka.pattern.AskTimeoutException on follower while BGP peer introduces 1M prefixes" class="issue-link" data-issue-key="CONTROLLER-1483"><del>CONTROLLER-1483</del></a> patch series so not sure what the exact<br/> &gt; behavior will be.</p> <p>what would be the downside to giving an "unavailable" response in such a <br/> case instead of waiting for a new leader?</p> <p>As I mentioned, it makes every reasonable effort to fulfill the request before returning an error. I don't think that's wrong and I don't think we should eliminate retries just to make tests run shorter. </p> <p>The election-timeout is shard-heartbeat-interval-in-millis (500) x shard-election-timeout-factor (20) which is 10 sec by default. For the test you can reduce the shard-election-timeout-factor in the .cfg file to say 10 to cut the time in half.</p> <p>The reason for retries is for resiliency. So if the leader was transitioning when a transaction was submitted, retries enable the transaction to succeed rather than failing immediately.</p> <p>This is a question of the programming model (as exposed to the application) versus implementation (as implemented by a particular data store).</p> <p>Exposing all transient states in the implementation (i.e. when elections are happening) will clutter all applications with boiler-plate copy&amp;pasted code, which needs to be consistent with internal implementation state machine &#8211; which can and will change in time.</p> <p>The <a href="https://jira.opendaylight.org/browse/CONTROLLER-1483" title="akka.pattern.AskTimeoutException on follower while BGP peer introduces 1M prefixes" class="issue-link" data-issue-key="CONTROLLER-1483"><del>CONTROLLER-1483</del></a> rework does not change things in this area (except perhaps having misaligned timers, which will need revision).</p> <p>The only way to address this is to have add an alternative read() method, where the user can specify an upper bound on the request response (and report an equivalent of TimeoutException). This in turn will need figuring out how to specify this is the RESTCONF request (i.e. define an ODL extension query parameter) and have the RESTCONF implementation use the constrained read method.</p> <p>We cannot reasonably support this for modifications, simply because we cannot guarantee convergence in arbitrary time interval. Hence if an application were to specify an upper bound on write-type operations, the results reported to the application would not necessarily be consistent with what actually happened.</p> <p>In any case, we are past offset-0 M5, which means we cannot really deliver this in Carbon.</p> <p>One more thing: 'upper bound' in this context is obviously not a hard deadline, but rather a wish. For variety of reasons implementations can only guarantee a response "soon after" the deadline.</p> <p>Actually, no change is needed in the APIs themselves: read() already returns a Future, hence all that is required is the extension and making RESTCONF pass an appropriate time constraint to get().</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=jluhrsen" class="user-hover" rel="jluhrsen">jluhrsen</a> this reads like the AAA issue we fixed in Fluorine. Is this still happening?</p> <p>That fix just pushes the delay to the restconf read, which will still take 20s w/o a leader. This will get worse with tell-based as I believe the timeout after retries is 2 min.</p> <p>well, I'm not sure we are on the same page then. The last time I looked at this, I guess, was 1.5 years ago? Anyway,<br/> going by my steps in the description I am seeing that there is no more response issue.</p> <p>1. kill the leader<br/> 2. kill another node<br/> 3. GET on /restconf/streams to the remaining node</p> <p>the result is a proper response (200) and data ( {"streams":{}} )</p> <p>does it jive with what you guys expect?</p> <p>That's b/c /restconf/streams doesn't hit the DS. You saw this before due to the AAA issue as Robert alluded to. However it would still happen for a request that does actually access the DS - that's expected.</p> <p>If there's no further action here then please close it.</p> <p>deal. nothing else here for me to worry about. closing.</p> <p>resolved with <a href="https://git.opendaylight.org/gerrit/#/c/73991/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/73991/</a></p> Development External issue ID 7892 External issue URL Issue Type Rank 0|i01y1z: