https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/CONTROLLER-1672 controller <p>Marking as a clustering bug, because this is not seen in 1node tests.</p> <p>Seen in BGP suite (if <a href="https://jira.opendaylight.org/browse/CONTROLLER-1645" title="shard moved during 1M bgp prefix advertizing (with tell-based=true)" class="issue-link" data-issue-key="CONTROLLER-1645">CONTROLLER-1645</a> does not come first). The URL is currently /restconf/modules, to be removed soon, but I think this may also happen when querying jolokia.</p> <p>The read has currently a timeout set to 5 seconds, which can be overstepped <span class="error">&#91;0&#93;</span>, but previous reads <span class="error">&#91;1&#93;</span> were also taking more than 4 seconds to finish. I see nothing critical in karaf.log considering this is tell-based protocol.</p> <p>Note that when testing locally, garbage collector was doing small collects only, so pauses under 1 second.</p> <p>This might be just a less severe version of 8432 (5s as opposed to 30s), or this might be a less severe version of <a href="https://jira.opendaylight.org/browse/CONTROLLER-1645" title="shard moved during 1M bgp prefix advertizing (with tell-based=true)" class="issue-link" data-issue-key="CONTROLLER-1645">CONTROLLER-1645</a> when the delay is not long enough to cause the leader movement due to transient dissociation.</p> <p>If this is not deemed critical, we can change suites to allow for slower responses.</p> <p><span class="error">&#91;0&#93;</span> <a href="https://logs.opendaylight.org/releng/jenkins092/bgpcep-csit-3node-periodic-bgpclustering-only-carbon/264/archives/log.html.gz#s1-s9-t8-k2-k3-k7-k2-k1-k6-k1-k1-k1-k1-k1-k2-k1-k1-k2-k4-k2-k1-k6-k3-k1-k2-k1-k1-k3-k3-k1" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/bgpcep-csit-3node-periodic-bgpclustering-only-carbon/264/archives/log.html.gz#s1-s9-t8-k2-k3-k7-k2-k1-k6-k1-k1-k1-k1-k1-k2-k1-k1-k2-k4-k2-k1-k6-k3-k1-k2-k1-k1-k3-k3-k1</a><br/> <span class="error">&#91;1&#93;</span> <a href="https://logs.opendaylight.org/releng/jenkins092/bgpcep-csit-3node-periodic-bgpclustering-only-carbon/264/archives/log.html.gz#s1-s9-t8-k2-k3-k7-k2-k1-k6-k1-k1-k1-k1-k1-k2-k1-k1-k2-k3-k2-k1-k6-k3-k1-k2-k1-k1-k3-k3-k1" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/bgpcep-csit-3node-periodic-bgpclustering-only-carbon/264/archives/log.html.gz#s1-s9-t8-k2-k3-k7-k2-k1-k6-k1-k1-k1-k1-k1-k2-k1-k1-k2-k3-k2-k1-k6-k3-k1-k2-k1-k1-k3-k3-k1</a></p> <p>Operating System: All<br/> Platform: All</p> CONTROLLER-1672

Restconf slow to respond when the member is under load

Bug Highest Resolved Cannot Reproduce Oleksii Mozghovyi Vratko Polak pt Fri, 12 May 2017 15:59:06 +0000 Tue, 25 Jul 2023 08:24:36 +0000 Fri, 2 Jul 2021 12:51:17 +0000 4.0.0 clustering 0 3 <p>/restconf/modules doesn't hit the datastore so if that is slow then the whole system is likely bogged down. With clustering, there's more load on the system due to replication etc. Could it be low on memory? CPU usage high? I would suggest tracking memory and CPU usage. If CPU is high, get thread dumps.</p> <p>Does this occur with odl-restconf-noauth? I think AAA has interactions with the data store, which would explain why it takes longer than expected.</p> <p>We hit this again <span class="error">&#91;2&#93;</span>, but few runs passed in the meantime. This time we do not access /restconf/modules but jolokia <span class="error">&#91;3&#93;</span> directly.</p> <p>&gt; Does this occur with odl-restconf-noauth</p> <p>I think on Sandbox it passed. We will switch the job to odl-restconf-noauth now.</p> <p><span class="error">&#91;2&#93;</span> <a href="https://logs.opendaylight.org/releng/jenkins092/bgpcep-csit-3node-periodic-bgpclustering-only-carbon/276/archives/log.html.gz#s1-s7-t8-k2-k3-k7-k4-k1-k6-k1-k1-k1-k1-k1-k2-k1-k1-k2-k3-k2-k1-k6-k2-k1-k5-k1-k3-k1" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/bgpcep-csit-3node-periodic-bgpclustering-only-carbon/276/archives/log.html.gz#s1-s7-t8-k2-k3-k7-k4-k1-k6-k1-k1-k1-k1-k1-k2-k1-k1-k2-k3-k2-k1-k6-k2-k1-k5-k1-k3-k1</a><br/> <span class="error">&#91;3&#93;</span> <a href="https://logs.opendaylight.org/releng/jenkins092/bgpcep-csit-3node-periodic-bgpclustering-only-carbon/276/archives/log.html.gz#s1-s7-t8-k2-k3-k7-k4-k1-k6-k1-k1-k1-k1-k1-k2-k1-k1-k2-k3-k2-k1-k6-k2-k1-k5-k1-k1" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/bgpcep-csit-3node-periodic-bgpclustering-only-carbon/276/archives/log.html.gz#s1-s7-t8-k2-k3-k7-k4-k1-k6-k1-k1-k1-k1-k1-k2-k1-k1-k2-k3-k2-k1-k6-k2-k1-k5-k1-k1</a></p> <p>&gt; switch the job to odl-restconf-noauth</p> <p>Done, but the jolokia read is still <span class="error">&#91;4&#93;</span> overstepping 5 seconds.</p> <p>This time we use tell-based protocol, rib owner and leaders were already on member-1 (no actual leader movement, neither by request nor Bug-8318).</p> <p>Karaf.log <span class="error">&#91;5&#93;</span> shows member-2 was (probably) installing snapshots, and there are warnings such as:<br/> 2017-05-17 11:21:20,929 | WARN | ult-dispatcher-4 | Shard | 192 - org.opendaylight.controller.sal-clustering-commons - 1.5.0.Carbon | member-2-shard-default-operational (Follower): Missing index 3062 from log. Cannot apply state. Ignoring 3062 to 3062<br/> 2017-05-17 11:21:25,620 | WARN | lt-dispatcher-29 | FrontendClientMetadataBuilder | 199 - org.opendaylight.controller.sal-distributed-datastore - 1.5.0.Carbon | member-2-shard-topology-operational: Unknown history for aborted transaction member-1-datastore-operational-fe-0-txn-7-1, ignoring</p> <p>I am not sure whether that explains why Jolokia is so slow. UnreachableMember has happened some time later after the test already failed:<br/> 2017-05-17 11:21:57,351 | INFO | lt-dispatcher-72 | ShardManager | 199 - org.opendaylight.controller.sal-distributed-datastore - 1.5.0.Carbon | Received UnreachableMember: memberName MemberName</p> {name=member-3} <p>, address: akka.tcp://opendaylight-cluster-data@10.29.12.21:2550</p> <p><span class="error">&#91;4&#93;</span> <a href="https://logs.opendaylight.org/releng/jenkins092/bgpcep-csit-3node-periodic-bgpclustering-only-carbon/279/archives/log.html.gz#s1-s2-t8-k2-k3-k7-k4-k1-k6-k1-k1-k1-k1-k1-k2-k1-k1-k2-k1-k2-k1-k6-k2-k1-k5-k1-k3-k1" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/bgpcep-csit-3node-periodic-bgpclustering-only-carbon/279/archives/log.html.gz#s1-s2-t8-k2-k3-k7-k4-k1-k6-k1-k1-k1-k1-k1-k2-k1-k1-k2-k1-k2-k1-k6-k2-k1-k5-k1-k3-k1</a><br/> <span class="error">&#91;5&#93;</span> <a href="https://logs.opendaylight.org/releng/jenkins092/bgpcep-csit-3node-periodic-bgpclustering-only-carbon/279/archives/odl2_karaf.log.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/bgpcep-csit-3node-periodic-bgpclustering-only-carbon/279/archives/odl2_karaf.log.gz</a></p> <p>&gt; jolokia read</p> <p>This can still happen. We were testing BGP longevity (tell-based protocol) on Sandbox, on a build which contained a chain <span class="error">&#91;0&#93;</span> for Carbon post-release with increased akka timers <span class="error">&#91;1&#93;</span>, and Jolokia did not respond within 30 seconds <span class="error">&#91;2&#93;</span>.</p> <p>The request was asking member-2 about state of default-operational shard (member-2 was a follower and rib owner).<br/> Here is the relevant segment of karaf.log <span class="error">&#91;3&#93;</span> , the request was open between 18:12:42.608 and 18:13:12.641</p> <p>2017-05-25 18:11:59,944 | WARN | lt-dispatcher-33 | Shard | 192 - org.opendaylight.controller.sal-clustering-commons - 1.5.0.SNAPSHOT | member-2-shard-default-operational (Follower): Missing index 242 from log. Cannot apply state. Ignoring 242 to 256<br/> 2017-05-25 18:12:11,601 | INFO | lt-dispatcher-77 | ShardManager | 199 - org.opendaylight.controller.sal-distributed-datastore - 1.5.0.SNAPSHOT | shard-manager-operational Received follower initial sync status for member-2-shard-default-operational status sync done true<br/> 2017-05-25 18:12:11,606 | INFO | lt-dispatcher-77 | ShardManager | 199 - org.opendaylight.controller.sal-distributed-datastore - 1.5.0.SNAPSHOT | shard-manager-operational Received follower initial sync status for member-2-shard-default-operational status sync done false<br/> 2017-05-25 18:12:25,496 | WARN | lt-dispatcher-33 | FrontendClientMetadataBuilder | 199 - org.opendaylight.controller.sal-distributed-datastore - 1.5.0.SNAPSHOT | member-2-shard-topology-operational: Unknown history for aborted transaction member-1-datastore-operational-fe-0-txn-6-2, ignoring<br/> 2017-05-25 18:12:25,523 | WARN | lt-dispatcher-33 | Shard | 192 - org.opendaylight.controller.sal-clustering-commons - 1.5.0.SNAPSHOT | member-2-shard-inventory-config (Leader) : handleAppendEntriesReply delayed beyond election timeout, appendEntriesReply : AppendEntriesReply <span class="error">&#91;term=1, success=true, followerId=member-1-shard-inventory-config, logLastIndex=-1, logLastTerm=-1, forceInstallSnapshot=false, payloadVersion=5, raftVersion=3&#93;</span>, timeSinceLastActivity : 16517, lastApplied : -1, commitIndex : -1<br/> 2017-05-25 18:12:25,523 | WARN | lt-dispatcher-27 | Shard | 192 - org.opendaylight.controller.sal-clustering-commons - 1.5.0.SNAPSHOT | member-2-shard-toaster-config (Leader) : handleAppendEntriesReply delayed beyond election timeout, appendEntriesReply : AppendEntriesReply <span class="error">&#91;term=1, success=true, followerId=member-1-shard-toaster-config, logLastIndex=-1, logLastTerm=-1, forceInstallSnapshot=false, payloadVersion=5, raftVersion=3&#93;</span>, timeSinceLastActivity : 16517, lastApplied : -1, commitIndex : -1<br/> 2017-05-25 18:12:25,523 | WARN | lt-dispatcher-82 | Shard | 192 - org.opendaylight.controller.sal-clustering-commons - 1.5.0.SNAPSHOT | member-2-shard-default-config (Leader) : handleAppendEntriesReply delayed beyond election timeout, appendEntriesReply : AppendEntriesReply <span class="error">&#91;term=1, success=true, followerId=member-1-shard-default-config, logLastIndex=68, logLastTerm=1, forceInstallSnapshot=false, payloadVersion=5, raftVersion=3&#93;</span>, timeSinceLastActivity : 16517, lastApplied : 68, commitIndex : 68<br/> 2017-05-25 18:12:25,523 | WARN | lt-dispatcher-32 | Shard | 192 - org.opendaylight.controller.sal-clustering-commons - 1.5.0.SNAPSHOT | member-2-shard-topology-operational (Leader) : handleAppendEntriesReply delayed beyond election timeout, appendEntriesReply : AppendEntriesReply <span class="error">&#91;term=1, success=true, followerId=member-1-shard-topology-operational, logLastIndex=58, logLastTerm=1, forceInstallSnapshot=false, payloadVersion=5, raftVersion=3&#93;</span>, timeSinceLastActivity : 22653, lastApplied : 59, commitIndex : 59<br/> 2017-05-25 18:12:25,524 | WARN | lt-dispatcher-79 | Shard | 192 - org.opendaylight.controller.sal-clustering-commons - 1.5.0.SNAPSHOT | member-2-shard-topology-config (Leader) : handleAppendEntriesReply delayed beyond election timeout, appendEntriesReply : AppendEntriesReply <span class="error">&#91;term=1, success=true, followerId=member-1-shard-topology-config, logLastIndex=5, logLastTerm=1, forceInstallSnapshot=false, payloadVersion=5, raftVersion=3&#93;</span>, timeSinceLastActivity : 16517, lastApplied : 5, commitIndex : 5<br/> 2017-05-25 18:12:25,525 | WARN | lt-dispatcher-33 | Shard | 192 - org.opendaylight.controller.sal-clustering-commons - 1.5.0.SNAPSHOT | member-2-shard-toaster-operational (Leader) : handleAppendEntriesReply delayed beyond election timeout, appendEntriesReply : AppendEntriesReply <span class="error">&#91;term=1, success=true, followerId=member-1-shard-toaster-operational, logLastIndex=-1, logLastTerm=-1, forceInstallSnapshot=false, payloadVersion=5, raftVersion=3&#93;</span>, timeSinceLastActivity : 16519, lastApplied : -1, commitIndex : -1<br/> 2017-05-25 18:12:25,595 | WARN | lt-dispatcher-33 | FrontendClientMetadataBuilder | 199 - org.opendaylight.controller.sal-distributed-datastore - 1.5.0.SNAPSHOT | member-2-shard-topology-operational: Unknown history for purged transaction member-1-datastore-operational-fe-0-txn-6-2, ignoring<br/> 2017-05-25 18:13:05,067 | WARN | lt-dispatcher-27 | ClusterCoreDaemon | 174 - com.typesafe.akka.slf4j - 2.4.17 | Cluster Node <span class="error">&#91;akka.tcp://opendaylight-cluster-data@10.29.12.69:2550&#93;</span> - Marking node(s) as UNREACHABLE <span class="error">&#91;Member(address = akka.tcp://opendaylight-cluster-data@10.29.12.236:2550, status = Up)&#93;</span>. Node roles <span class="error">&#91;member-2&#93;</span></p> <p>So even with increased timers <span class="error">&#91;1&#93;</span>, <a href="https://jira.opendaylight.org/browse/CONTROLLER-1645" title="shard moved during 1M bgp prefix advertizing (with tell-based=true)" class="issue-link" data-issue-key="CONTROLLER-1645">CONTROLLER-1645</a> can still happen.<br/> The question is whether UnreachableMember can cause Jolokia to respond slowly. One possibility is AAA sill storing something in the datastore, even though this run was using odl-restconf-noauth. Another possibility is shard access being blocked under load for some reason.</p> <p><span class="error">&#91;0&#93;</span> <a href="https://git.opendaylight.org/gerrit/#/c/57822/1" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/57822/1</a><br/> <span class="error">&#91;1&#93;</span> <a href="https://git.opendaylight.org/gerrit/#/c/57699/5" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/57699/5</a><br/> <span class="error">&#91;2&#93;</span> <a href="https://logs.opendaylight.org/sandbox/jenkins091/bgpcep-csit-3node-bgpclustering-longevity-only-carbon/13/archives/log.html.gz#s1-s2-t1-k9-k1-k1-k1-k1-k1-k1-k1-k1-k1-k2-k1-k3-k7-k5-k1-k6-k1-k1-k1-k1-k1-k2-k1-k1-k2-k2-k2-k1-k6-k2-k1-k5-k1-k3-k1" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/sandbox/jenkins091/bgpcep-csit-3node-bgpclustering-longevity-only-carbon/13/archives/log.html.gz#s1-s2-t1-k9-k1-k1-k1-k1-k1-k1-k1-k1-k1-k2-k1-k3-k7-k5-k1-k6-k1-k1-k1-k1-k1-k2-k1-k1-k2-k2-k2-k1-k6-k2-k1-k5-k1-k3-k1</a><br/> <span class="error">&#91;3&#93;</span> <a href="https://logs.opendaylight.org/sandbox/jenkins091/bgpcep-csit-3node-bgpclustering-longevity-only-carbon/13/archives/odl1_karaf.log.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/sandbox/jenkins091/bgpcep-csit-3node-bgpclustering-longevity-only-carbon/13/archives/odl1_karaf.log.gz</a></p> <p>The request to get the Shard stats sends a message to the Shard so it will wait in the queue and can get delayed. That's probably what's happening.</p> <p>Tom, I think we can improve the situation now that we have <a href="https://jira.opendaylight.org/browse/AAA-99" title="Upgrade Jersey from 1.X to 2.X" class="issue-link" data-issue-key="AAA-99"><del>AAA-99</del></a> addressed by adding making shard stats message also a control message. Do you agree?</p> <p>sure</p> <p>This looks like pretty old issue, so I had to fix several issues before taking care of RESTconf performance:<br/> 1. The 'odl-restconf-noauth' feature was removed in 2019; as a temporary solution, it can be replaced with the 'odl-restconf' feature. We would need to replace it with 'odl-restconf-nb-rfc8040' and do not install unnecessary dependencies for the test in a long-term perspective.<br/> <a href="https://git.opendaylight.org/gerrit/c/releng/builder/+/95526" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/c/releng/builder/+/95526</a><br/> 2. After running the test scenario multiple times in the clustered configuration, I noticed, that periodically the BGPCEP component is trying to instantiate the BgpPeer instances twice. Such behavior leads to the example-bgp-rib-service-group service failure. I didn't find the root cause of that behavior yet, but I could fix it by adding a small check to the BgpPeer class.<br/> <a href="https://git.opendaylight.org/gerrit/c/bgpcep/+/95561" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/c/bgpcep/+/95561</a></p> <p>3. On the sandbox I was able to run stability test for 3h with 6Gb of RAM allocated for controller. I've scheduled another run for 23h (original value for this particular test) and will check logs once its done.</p> <p>23H run on a sandbox with 6Gb of RAM allocated finished successfully; I will try to reduce memory to 4Gb and see what happens(on my local environment it failed).</p> <p>23H run on sandbox with 4Gb of RAM allocated finished successfully; the only problem is that size of logs files generated doesn't allow to push them into nexus. However, this can't be closed until the # 95561 change is merged to the bgpcep project.</p> <p>We cannot reproduce this issue with 4G heap, hence resolved.</p> Blocks AAA-99 BGPCEP-948 Development External issue ID 8434 External issue URL Issue Type ODL SR Target Milestone Rank 0|i02s0n: