https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/CONTROLLER-1724 controller <p>This is the robot symptom part of <a href="https://jira.opendaylight.org/browse/CONTROLLER-1711" title="Listener registration lost when local replica is removed" class="issue-link" data-issue-key="CONTROLLER-1711"><del>CONTROLLER-1711</del></a>. As discussed there, one possible reason is the test listener not waiting for quiescence, proposed fix is here <span class="error">&#91;0&#93;</span>.</p> <p>I am opening this because we have changed the test suite to not remove shard replicas, instead cluster-admin:make-leader-local is used to force leader movement, so the recent inconsistency <span class="error">&#91;1&#93;</span> is not caused by <a href="https://jira.opendaylight.org/browse/CONTROLLER-1711" title="Listener registration lost when local replica is removed" class="issue-link" data-issue-key="CONTROLLER-1711"><del>CONTROLLER-1711</del></a>.</p> <p><span class="error">&#91;0&#93;</span> <a href="https://git.opendaylight.org/gerrit/58952" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/58952</a><br/> <span class="error">&#91;1&#93;</span> <a href="https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/759/log.html.gz#s1-s36-t1-k2-k14-k2-k1-k4-k7-k1" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/759/log.html.gz#s1-s36-t1-k2-k14-k2-k1-k4-k7-k1</a></p> <p>Operating System: All<br/> Platform: All</p> CONTROLLER-1724

Robot test for data change listener reports data inconsistency

Bug Confirmed Unresolved Vratko Polak Vratko Polak Fri, 23 Jun 2017 14:14:43 +0000 Tue, 25 Jul 2023 08:24:44 +0000 clustering 0 4 <p>Reopening for investigation.</p> <p>A patch set which fixes other bugs is reintroducing this bug. See Sandbox <span class="error">&#91;3&#93;</span>.</p> <p><span class="error">&#91;2&#93;</span> <a href="https://git.opendaylight.org/gerrit/#/c/60137/4" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/60137/4</a><br/> <span class="error">&#91;3&#93;</span> <a href="https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-ls-only-carbon/4/log.html.gz#s1-s4-t1-k2-k14-k2-k1-k4-k7-k1" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-ls-only-carbon/4/log.html.gz#s1-s4-t1-k2-k14-k2-k1-k4-k7-k1</a></p> <p>There is a work in progress to improve the listener unsubscribe and verify logic. Current patch set <span class="error">&#91;4&#93;</span> fails to respond within 125 seconds <span class="error">&#91;5&#93;</span>, but when the timeout is increased, it results in "Connection aborted" <span class="error">&#91;6&#93;</span>. The huge karaf.log is here <span class="error">&#91;7&#93;</span>.</p> <p><span class="error">&#91;4&#93;</span> <a href="https://git.opendaylight.org/gerrit/#/c/60270/10" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/60270/10</a><br/> <span class="error">&#91;5&#93;</span> <a href="https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-ls-only-carbon/1/log.html.gz#s1-s2-t1-k2-k14-k2-k1-k4-k6-k1" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-ls-only-carbon/1/log.html.gz#s1-s2-t1-k2-k14-k2-k1-k4-k6-k1</a><br/> <span class="error">&#91;6&#93;</span> <a href="https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-ls-only-carbon/2/log.html.gz#s1-s2-t1-k2-k14-k2-k1-k4-k6-k1" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-ls-only-carbon/2/log.html.gz#s1-s2-t1-k2-k14-k2-k1-k4-k6-k1</a><br/> <span class="error">&#91;7&#93;</span> <a href="https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-ls-only-carbon/2/odl2_karaf.log.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-ls-only-carbon/2/odl2_karaf.log.gz</a></p> <p>&gt; "Connection aborted"</p> <p>Oh, I forgot trace verbosity leads to memory exhaustion.<br/> Will repeat without trace.</p> <p>&gt; Current patch set <span class="error">&#91;4&#93;</span> fails to respond within</p> <p>... 600 seconds <span class="error">&#91;8&#93;</span>. </p> <p>&gt; The huge karaf.log is</p> <p>... here <span class="error">&#91;9&#93;</span>.</p> <p><span class="error">&#91;8&#93;</span> <a href="https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-ls-only-carbon/4/log.html.gz#s1-s2-t1-k2-k14-k2-k1-k4-k6-k1" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-ls-only-carbon/4/log.html.gz#s1-s2-t1-k2-k14-k2-k1-k4-k6-k1</a><br/> <span class="error">&#91;9&#93;</span> <a href="https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-ls-only-carbon/4/odl2_karaf.log.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-ls-only-carbon/4/odl2_karaf.log.gz</a></p> <p>I think the problem comes from org.opendaylight.mdsal.dom.broker.ShardedDOMDataTree and how ShardedDOMDataTreeListenerContext processes notifications.</p> <p>That is to say the receive path is too slow to keep up, creating a long tail. The implementation needs to be revisited and optimized.</p> <p>Sandbox with this <span class="error">&#91;10&#93;</span> code, huge log <span class="error">&#91;11&#93;</span> contains this segment, which may be pointing to a cause why the test fail.</p> <p>2017-08-03 09:01:28,837 | DEBUG | pool-31-thread-1 | AbstractTransactionHandler | 257 - org.opendaylight.control<br/> ler.samples.clustering-it-provider - 1.5.2.SNAPSHOT | New future #39998 allocated<br/> 2017-08-03 09:01:28,838 | DEBUG | pool-31-thread-1 | AbstractTransactionHandler | 257 - org.opendaylight.control<br/> ler.samples.clustering-it-provider - 1.5.2.SNAPSHOT | Reached maximum run time with 2 outstanding futures<br/> 2017-08-03 09:01:28,840 | DEBUG | pool-31-thread-1 | AbstractTransactionHandler | 257 - org.opendaylight.control<br/> ler.samples.clustering-it-provider - 1.5.2.SNAPSHOT | Reached maximum run time with 2 outstanding futures<br/> 2017-08-03 09:01:28,840 | DEBUG | pool-31-thread-1 | AbstractTransactionHandler | 257 - org.opendaylight.control<br/> ler.samples.clustering-it-provider - 1.5.2.SNAPSHOT | Reached maximum run time with 2 outstanding futures<br/> 2017-08-03 09:01:28,847 | DEBUG | on-dispatcher-59 | AbstractDataListener | 257 - org.opendaylight.control<br/> ler.samples.clustering-it-provider - 1.5.2.SNAPSHOT | Received 1 data tree changed events<br/> 2017-08-03 09:01:28,855 | DEBUG | on-dispatcher-68 | AbstractDataListener | 257 - org.opendaylight.control<br/> ler.samples.clustering-it-provider - 1.5.2.SNAPSHOT | Received 1 data tree changed events<br/> 2017-08-03 09:01:28,856 | DEBUG | on-dispatcher-59 | AbstractDataListener | 257 - org.opendaylight.control<br/> ler.samples.clustering-it-provider - 1.5.2.SNAPSHOT | Received 1 data tree changed events<br/> 2017-08-03 09:01:28,890 | WARN | lt-dispatcher-41 | ShardedDataTreeActor | 199 - org.opendaylight.control<br/> ler.sal-distributed-datastore - 1.5.2.SNAPSHOT | The notification contained a path on which no producer is registered,<br/> throwing away<br/> 2017-08-03 09:01:28,902 | WARN | lt-dispatcher-41 | ShardedDataTreeActor | 199 - org.opendaylight.control<br/> ler.sal-distributed-datastore - 1.5.2.SNAPSHOT | The notification contained a path on which no producer is registered,<br/> throwing away<br/> 2017-08-03 09:01:28,908 | WARN | lt-dispatcher-41 | ShardedDataTreeActor | 199 - org.opendaylight.control<br/> ler.sal-distributed-datastore - 1.5.2.SNAPSHOT | The notification contained a path on which no producer is registered,<br/> throwing away<br/> 2017-08-03 09:01:28,911 | WARN | lt-dispatcher-41 | ShardedDataTreeActor | 199 - org.opendaylight.control<br/> ler.sal-distributed-datastore - 1.5.2.SNAPSHOT | The notification contained a path on which no producer is registered,<br/> throwing away<br/> 2017-08-03 09:01:28,914 | WARN | lt-dispatcher-41 | ShardedDataTreeActor | 199 - org.opendaylight.control<br/> ler.sal-distributed-datastore - 1.5.2.SNAPSHOT | The notification contained a path on which no producer is registered,<br/> throwing away<br/> 2017-08-03 09:01:28,916 | WARN | lt-dispatcher-41 | ShardedDataTreeActor | 199 - org.opendaylight.control<br/> ler.sal-distributed-datastore - 1.5.2.SNAPSHOT | The notification contained a path on which no producer is registered,<br/> throwing away<br/> 2017-08-03 09:01:28,919 | WARN | lt-dispatcher-41 | ShardedDataTreeActor | 199 - org.opendaylight.control<br/> ler.sal-distributed-datastore - 1.5.2.SNAPSHOT | The notification contained a path on which no producer is registered,<br/> throwing away<br/> 2017-08-03 09:01:28,922 | WARN | lt-dispatcher-41 | ShardedDataTreeActor | 199 - org.opendaylight.control<br/> ler.sal-distributed-datastore - 1.5.2.SNAPSHOT | The notification contained a path on which no producer is registered,<br/> throwing away<br/> 2017-08-03 09:01:28,924 | WARN | lt-dispatcher-41 | ShardedDataTreeActor | 199 - org.opendaylight.control<br/> ler.sal-distributed-datastore - 1.5.2.SNAPSHOT | The notification contained a path on which no producer is registered,<br/> throwing away<br/> 2017-08-03 09:01:28,927 | WARN | lt-dispatcher-41 | ShardedDataTreeActor | 199 - org.opendaylight.control<br/> ler.sal-distributed-datastore - 1.5.2.SNAPSHOT | The notification contained a path on which no producer is registered,<br/> throwing away<br/> 2017-08-03 09:01:28,930 | WARN | lt-dispatcher-41 | ShardedDataTreeActor | 199 - org.opendaylight.control<br/> ler.sal-distributed-datastore - 1.5.2.SNAPSHOT | The notification contained a path on which no producer is registered,<br/> throwing away<br/> 2017-08-03 09:01:28,933 | WARN | lt-dispatcher-41 | ShardedDataTreeActor | 199 - org.opendaylight.control<br/> ler.sal-distributed-datastore - 1.5.2.SNAPSHOT | The notification contained a path on which no producer is registered,<br/> throwing away<br/> 2017-08-03 09:01:28,935 | WARN | lt-dispatcher-41 | ShardedDataTreeActor | 199 - org.opendaylight.control<br/> ler.sal-distributed-datastore - 1.5.2.SNAPSHOT | The notification contained a path on which no producer is registered,<br/> throwing away<br/> 2017-08-03 09:01:28,937 | WARN | lt-dispatcher-41 | ShardedDataTreeActor | 199 - org.opendaylight.control<br/> ler.sal-distributed-datastore - 1.5.2.SNAPSHOT | The notification contained a path on which no producer is registered,<br/> throwing away<br/> 2017-08-03 09:01:28,940 | WARN | lt-dispatcher-41 | ShardedDataTreeActor | 199 - org.opendaylight.control<br/> ler.sal-distributed-datastore - 1.5.2.SNAPSHOT | The notification contained a path on which no producer is registered,<br/> throwing away<br/> 2017-08-03 09:01:29,849 | DEBUG | on-dispatcher-62 | AbstractDataListener | 257 - org.opendaylight.control<br/> ler.samples.clustering-it-provider - 1.5.2.SNAPSHOT | Received 1 data tree changed events</p> <p><span class="error">&#91;10&#93;</span> <a href="https://git.opendaylight.org/gerrit/#/c/60270/26" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/60270/26</a><br/> <span class="error">&#91;11&#93;</span> <a href="https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-ls-only-carbon/1/odl2_karaf.log.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-ls-only-carbon/1/odl2_karaf.log.gz</a></p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=vrpolak" class="user-hover" rel="vrpolak">vrpolak</a> is this still reproducible?</p> <p>Yes, still reproducible. Not every run contains the Robot failure, but they are frequent enough.<br/> Recent case of odl-mdsal-lowlevel-control:unsubscribe-ddtl response with copy-matches being false is here: <span class="error">&#91;12&#93;</span>.</p> <p><span class="error">&#91;12&#93;</span> <a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/controller-csit-3node-clustering-tell-all-neon/67/robot-plugin/log.html.gz#s1-s24-t3-k2-k16" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/controller-csit-3node-clustering-tell-all-neon/67/robot-plugin/log.html.gz#s1-s24-t3-k2-k16</a></p> Development External issue ID 8733 External issue URL Issue Type Rank 0|i02sc7: