OpenDaylight JIRA

OpenDaylight JIRA https://jira.opendaylight.org This file is an XML representation of an issue en-us

8.20.10

820010

22-06-2022

[CONTROLLER-1569] B and C: ds benchmark unstable for READ and DELETE operation in 3node cluster https://jira.opendaylight.org/browse/CONTROLLER-1569 controller <p>1node ds benchmark jobs dont show unstability</p> <p><a href="https://jenkins.opendaylight.org/releng/view/controller/job/controller-csit-1node-periodic-benchmark-all-boron/plot/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/controller/job/controller-csit-1node-periodic-benchmark-all-boron/plot/</a><br/> <a href="https://jenkins.opendaylight.org/releng/view/controller/job/controller-csit-1node-periodic-benchmark-all-carbon/plot/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/controller/job/controller-csit-1node-periodic-benchmark-all-carbon/plot/</a><br/> <a href="https://jenkins.opendaylight.org/releng/view/controller/job/controller-csit-1node-periodic-benchmark-only-boron/plot/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/controller/job/controller-csit-1node-periodic-benchmark-only-boron/plot/</a><br/> <a href="https://jenkins.opendaylight.org/releng/view/controller/job/controller-csit-1node-periodic-benchmark-only-carbon/plot/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/controller/job/controller-csit-1node-periodic-benchmark-only-carbon/plot/</a></p> <p>and all operations (READ,PUT,MERGE,DELETE) preforms stable according to the long term thrends</p> <p>3node ds benchmark jobs show that PUT and MERGE preform stable, but READ and DELETE dont</p> <p><a href="https://jenkins.opendaylight.org/releng/view/controller/job/controller-csit-3node-periodic-benchmark-all-boron/plot/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/controller/job/controller-csit-3node-periodic-benchmark-all-boron/plot/</a><br/> <a href="https://jenkins.opendaylight.org/releng/view/controller/job/controller-csit-3node-periodic-benchmark-all-carbon/plot/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/controller/job/controller-csit-3node-periodic-benchmark-all-carbon/plot/</a><br/> <a href="https://jenkins.opendaylight.org/releng/view/controller/job/controller-csit-3node-periodic-benchmark-only-boron/plot/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/controller/job/controller-csit-3node-periodic-benchmark-only-boron/plot/</a><br/> <a href="https://jenkins.opendaylight.org/releng/view/controller/job/controller-csit-3node-periodic-benchmark-only-carbon/plot/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/controller/job/controller-csit-3node-periodic-benchmark-only-carbon/plot/</a></p> <p>Having a closer look at e.g.<br/> <a href="https://jenkins.opendaylight.org/releng/view/controller/job/controller-csit-3node-periodic-benchmark-only-carbon/plot/getPlot?index=1&width=750&height=1000" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/controller/job/controller-csit-3node-periodic-benchmark-only-carbon/plot/getPlot?index=1&width=750&height=1000</a><br/> shows that ds benchmark with FOL_(running on a follower node) prefix are not that stable</p> <p>Operating System: All<br/> Platform: All</p> CONTROLLER-1569

B and C: ds benchmark unstable for READ and DELETE operation in 3node cluster

Bug Resolved Done Robert Varga Peter Gubka Thu, 15 Dec 2016 13:28:08 +0000 Tue, 25 Jul 2023 08:24:16 +0000 Mon, 8 May 2017 13:07:07 +0000 clustering 0 2 <p>in the past i wrote an email for this problem too<br/> <a href="https://www.mail-archive.com/controller-dev@lists.opendaylight.org/msg00461.html" class="external-link" target="_blank" rel="nofollow noopener">https://www.mail-archive.com/controller-dev@lists.opendaylight.org/msg00461.html</a></p> <p>Interesting fluctuations. It looks like we have a battle between leader and follower, visible from:</p> <p><a href="https://jenkins.opendaylight.org/releng/view/controller/job/controller-csit-3node-periodic-benchmark-only-carbon/plot/getPlot?index=4&width=1750&height=1450" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/controller/job/controller-csit-3node-periodic-benchmark-only-carbon/plot/getPlot?index=4&width=1750&height=1450</a></p> <p>Note green median (CONFIG) and the wide disconnect between blue and purple set of lines. I think their average is slightly above the config line.</p> <p>Is it possible that 'follower' and 'leader' are really talking to the shard leader?</p> <p>Also, could we get the Y-axis show operations per millisecond, i.e. 100K ops in 200ms == 500ops/ms?</p> <p>I think this needs some profiling to understand what is going on.</p> <p>(In reply to Robert Varga from comment #2)<br/> > Interesting fluctuations. It looks like we have a battle between leader and<br/> > follower, visible from:<br/> > <br/> > <a href="https://jenkins.opendaylight.org/releng/view/controller/job/controller-csit-" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/controller/job/controller-csit-</a><br/> > 3node-periodic-benchmark-only-carbon/plot/<br/> > getPlot?index=4&width=1750&height=1450<br/> > <br/> > Note green median (CONFIG) and the wide disconnect between blue and purple<br/> > set of lines. I think their average is slightly above the config line.<br/> > <br/> > Is it possible that 'follower' and 'leader' are really talking to the shard<br/> > leader?</p> <p>From the log.html it can be seen that before the python bechmarch script starts, the robot iterates over all 3 nodes and gets information about the shard (e.g. from jolokia/read/org.opendaylight.controller:Category=Shards,name=member-1-shard-default-config,type=DistributedConfigDatastore)<br/> Then runs the python script against the leader's or follower's ip address. There is no check during the test about the leader movement.</p> <p>> <br/> > Also, could we get the Y-axis show operations per millisecond, i.e. 100K ops<br/> > in 200ms == 500ops/ms?</p> <p>Is this a request to modify the job? In this case we will either loose history and present graphs or the lines will be plotted in present graphs, but it can be very ugly.</p> <p>As csv files are stored for every jenkins run, i can create a post-processing script (if really needed) which will recount/create new csv files with new Y-axis values. It will be run locally and graphs will have to be also created somehow locally.</p> <p>> <br/> > I think this needs some profiling to understand what is going on.</p> <p>Attachment b211_odl3.log has been added with description: Measure_Leader_Operational_Txchain_Read log from #211</p> <p>Attachment b212_odl1.log has been added with description: Measure_Leader_Operational_Txchain_Read log from #212</p> <p>Since the build #210 the job is back to the blue dot.</p> <p>Now we have 3 builds with blue dots.<br/> <a href="https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-periodic-benchmark-only-carbon/210/" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-periodic-benchmark-only-carbon/210/</a><br/> <a href="https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-periodic-benchmark-only-carbon/211/" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-periodic-benchmark-only-carbon/211/</a><br/> <a href="https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-periodic-benchmark-only-carbon/212/" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-periodic-benchmark-only-carbon/212/</a></p> <p>The robot's log.html details shows that each test case lasts 3-6 minutes. There are exceptions for the READ tests which either lasts usual time or the last up to 35 minuts.</p> <p>It does not affect follower's tests only. </p> <p>E.g. test case duration of </p> <p>Measure_Leader_Operational_Txchain_Read<br/> #210 32:52<br/> #211 2:51<br/> #212 30:11</p> <p>Measure_Follower_Operational_Simpletx_Read<br/> #211 35:32<br/> #212 5:00</p> <p>Checking logs on leaders for Measure_Leader_Operational_Txchain_Read i fount that a good tc from #211 did not contain any messages like FrontendClientMetadataBuilder | 206 - org.opendaylight.controller.sal-distributed-datastore - 1.5.0.SNAPSHOT | Unknown history for aborted transaction member-1-datastore-config-fe-0-txn-127-0, ignoring</p> <p>Logs attached.</p> <p>I'm not familiar with the tests you're running but if the test times are fluctuating on jenkins runs perhaps it's due to instability in the upstream build environment. I would suggest you run the tests multiple times on local systems and compare.</p> <p>(In reply to Tom Pantelis from comment #7)<br/> > I'm not familiar with the tests you're running but if the test times are<br/> > fluctuating on jenkins runs perhaps it's due to instability in the upstream<br/> > build environment. I would suggest you run the tests multiple times on local<br/> > systems and compare.</p> <p>Only READ tests have problem. PUT, MERGE and DELETE look ok. I would prefer some hints how to debug the READ problem than reproducing all tests locally.</p> <p>Perhaps the reads are being performed remotely when they're intended to be local or the node that is selected could be either a follower or a leader? Remote reads from a follower will be significantly slower than a local read. Also the size of the data could be significant for remote reads as it needs to ser/des and transport it over the wire. Remote reads are largely dependent on the speed of the network.</p> <p>(In reply to Peter Gubka from comment #8)<br/> > (In reply to Tom Pantelis from comment #7)<br/> > > I'm not familiar with the tests you're running but if the test times are<br/> > > fluctuating on jenkins runs perhaps it's due to instability in the upstream<br/> > > build environment. I would suggest you run the tests multiple times on local<br/> > > systems and compare.<br/> > <br/> > Only READ tests have problem. PUT, MERGE and DELETE look ok. I would prefer<br/> > some hints how to debug the READ problem than reproducing all tests locally.</p> <p>Checking <a href="https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-periodic-benchmark-only-carbon/229/archives/" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-periodic-benchmark-only-carbon/229/archives/</a><br/> there was no leader change during the test. All changes are done before suite starts. Same for jobs #225 and #226.</p> <p>I have audited dsbenchmark and the fact is that its results with operational store are irrelevant due to inconsistencies in setup (which writes into operational) and actual execution (which is hard-wired to CONFIG).</p> <p>carbon: <a href="https://git.opendaylight.org/gerrit/56108" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/56108</a></p> <p>We'll need to see what the results will look like.</p> Development External issue ID 7390 External issue URL Issue Type Rank 0|i02rdr: