https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/CONTROLLER-1711 controller <p>This was seen on Sandbox on a code which is already merged to stable/carbon.</p> <p>The title says possibly, as from the two similar test cases, only one has failed <span class="error">&#91;0&#93;</span>, it was the one where the listener was on the same member as the shard leader, which has been moved by calling remove-prefix-shard-replica.</p> <p>In huge karaf.log <span class="error">&#91;1&#93;</span> with some debugs look between removal starting at 19:23:10,608 and suite adding replica back at 19:23:41,773.</p> <p><span class="error">&#91;0&#93;</span> <a href="https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-only-carbon/6/log.html.gz#s1-s20-t1-k2-k16-k2-k1-k4-k7-k1" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-only-carbon/6/log.html.gz#s1-s20-t1-k2-k16-k2-k1-k4-k7-k1</a><br/> <span class="error">&#91;1&#93;</span> <a href="https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-only-carbon/6/odl3_karaf.log.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-only-carbon/6/odl3_karaf.log.gz</a></p> <p>Operating System: All<br/> Platform: All</p> CONTROLLER-1711

Listener registration lost when local replica is removed

Bug Resolved Won't Do Unassigned Vratko Polak Wed, 7 Jun 2017 16:39:31 +0000 Tue, 25 Jul 2023 08:24:42 +0000 Fri, 5 Oct 2018 15:41:22 +0000 clustering 0 3 <p>So from my understanding, a DTCL is registered for shard A on member1 then shard A is removed from member1. Later shard A is re-added to member1 and the DTCL is not notified. Is this the scenario? If so that's to be expected as the DTCL registrations belong to the shard and thus "go away" when the shard does. Kind of an edge case...</p> <p>Now seen on RelEng <span class="error">&#91;5&#93;</span>.</p> <p>&gt; Later shard A is re-added to member1</p> <p>That is only done in test teardown, so that is not part of the scenario.</p> <p>&gt; DTCL is registered for shard A on member1 then shard A is removed from member1.</p> <p>This a DDTL <span class="error">&#91;2&#93;</span>, not DTCL.</p> <p>If the listener is on a follower, removal of the leader shard replica does not lead to failures <span class="error">&#91;3&#93;</span>, thus the new leader continues to send notifications without missing any (or the new leader does not commit anything, which I would expect to lead to errors in producers <span class="error">&#91;4&#93;</span>).</p> <p><span class="error">&#91;2&#93;</span> <a href="https://github.com/opendaylight/controller/blob/5997e14efab9c12e7be2b7fb83f7efe16c2bfe7c/opendaylight/md-sal/samples/clustering-test-app/provider/src/main/java/org/opendaylight/controller/clustering/it/provider/impl/IdIntsDOMDataTreeLIstener.java#L23" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/opendaylight/controller/blob/5997e14efab9c12e7be2b7fb83f7efe16c2bfe7c/opendaylight/md-sal/samples/clustering-test-app/provider/src/main/java/org/opendaylight/controller/clustering/it/provider/impl/IdIntsDOMDataTreeLIstener.java#L23</a><br/> <span class="error">&#91;3&#93;</span> <a href="https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/737/log.html.gz#s1-s38-t3-k2-k16-k2-k1-k4-k7-k1" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/737/log.html.gz#s1-s38-t3-k2-k16-k2-k1-k4-k7-k1</a><br/> <span class="error">&#91;4&#93;</span> <a href="https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/737/log.html.gz#s1-s38-t3-k2-k14" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/737/log.html.gz#s1-s38-t3-k2-k14</a><br/> <span class="error">&#91;5&#93;</span> <a href="https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/737/log.html.gz#s1-s38-t1-k2-k16-k2-k1-k4-k7-k1" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/737/log.html.gz#s1-s38-t1-k2-k16-k2-k1-k4-k7-k1</a></p> <p>Sandbox testing results.<br/> Transaction producers confirmed successful end at 11:41:45.908 <span class="error">&#91;6&#93;</span><br/> The unsubscribe-ddtl has been called between 11:41:46.006 and 11:41:47.293 <span class="error">&#91;7&#93;</span>, but the (huge) karaf.log <span class="error">&#91;8&#93;</span> shows a change being received as late as 09:42:03,183, just because the member has been restarted at that point <span class="error">&#91;9&#93;</span>.</p> <p>Is there a reasonable way to wait until no notification is being processed?</p> <p><span class="error">&#91;6&#93;</span> <a href="https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-only-carbon/21/log.html.gz#s1-s4-t1-k2-k14" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-only-carbon/21/log.html.gz#s1-s4-t1-k2-k14</a><br/> <span class="error">&#91;7&#93;</span> <a href="https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-only-carbon/21/log.html.gz#s1-s4-t1-k2-k16-k2-k1-k4-k6-k1" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-only-carbon/21/log.html.gz#s1-s4-t1-k2-k16-k2-k1-k4-k6-k1</a><br/> <span class="error">&#91;8&#93;</span> <a href="https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-only-carbon/21/odl3_karaf.log.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-only-carbon/21/odl3_karaf.log.gz</a><br/> <span class="error">&#91;9&#93;</span> <a href="https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-only-carbon/21/log.html.gz#s1-s4-t2-k2-k1-k3-k2-k3-k1-k2-k2-k1-k2-k1-k6" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-only-carbon/21/log.html.gz#s1-s4-t2-k2-k1-k3-k2-k3-k1-k2-k2-k1-k2-k1-k6</a></p> <p>&gt; the new leader continues to send notifications without missing any</p> <p>Okay, so this is where I was wrong. The current cluster-wide listener implementation apparently relies on local replica data. No local replica, no data change notifications.</p> <p>Since this Bug now tracks the missing functionality, I will open another one for listener failures not related to missing local replica.</p> Development External issue ID 8629 External issue URL Issue Type Rank 0|i02s9b: