https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/CONTROLLER-1399 controller <p>Test Environment : Single Node Controller</p> <p>For following case, DCN is received in normal operational condition and no DCN is received when the node gets restarted (ie. restoration is completed). </p> <p>Checked if the registration for DCN happened consistently during restart and registration seems to have no issues</p> <p>Normal Behavior </p> <p>Path Scope Data Change DCN Received <br/> ========================================================================<br/> /container/list BASE Add entry to list On per entry basis <br/> /container/list SUBTREE Add entry to list On per entry basis </p> <p>Restart Behavior</p> <p>Path Scope Data Change DCN Received <br/> ========================================================================<br/> /container/list BASE Add entry to list No DCN event Received <br/> /container/list SUBTREE Add entry to list No DCN event Received </p> <p>When normal DCN behavior is different from restart behavior apps which act on DCN behave differently when nodal restart occurs</p> <p>Operating System: All<br/> Platform: All</p> CONTROLLER-1399

Clustering: DCN varies in normal and restart scenarios

Bug Verified Done Tom Pantelis Muthukumaran Kothandaraman Wed, 5 Aug 2015 09:28:34 +0000 Mon, 14 Sep 2015 14:30:55 +0000 Mon, 14 Sep 2015 14:30:55 +0000 Lithium clustering 0 12 <p>Please note that during restart, no entry is created newly in the mentioned path. Instead data is restored from persistent store. </p> <p>The expectation is that the DCN should fire consistently in normal and restart conditions provided path and scope of registration remains the same</p> <p>Attachment Controller restart test case and logs.txt has been added with description: Controller Restart Test Case, Log And Analysis</p> <p>The bug is valid even in normal operation condition, where the listener registers late for DCNs and before listener got registered, data is available in the datastore.</p> <p>Are you using DataChangeListener interface or DataTreeChangeListener interface?<br/> Are you registering your listener using wildcards (not specifying keys?)</p> <p>Tony, we are using DataChangeListener interface.</p> <p>The listener registration is wildcard. The app is interested in all list entries.</p> <p>Hi Tony, </p> <p>Is firing DCNs during recovery of Config DS is a requirement irrespective of this bug ?</p> <p>This can introduce issues around scaling. For instance, when a specific subtree recovers a million entries and put back to Config DS, if DCNs are fired and apps take their own time executing their side-effects in DCNs, would that not cause unpredictable delays in recovery ?</p> <p>Regards<br/> Muthu</p> <p>Hi Tony, </p> <p>Already clustered datastore is doing a deferred Listener registration after the recovery is completed. So, ideally, the notifications should not fire at all. </p> <p>As mentioned by Deepthi, we also tried late-registration - ie. populate datastore with few entries and then perform listener registration. In this case too we do not get notifications. </p> <p>So, essentially the question boils-down to the exact expected behavior during recovery so that apps do not follow usage patterns like <br/> transact with Config DS -&gt; get DCN -&gt; transact with Operational DS.</p> <p>Such app patterns can lead to troubles during restart. </p> <p>Moreover, playing DCNs during recovery is actually a bad pattern.</p> <p>What are your thoughts ?</p> <p> final Optional&lt;NormalizedNode&lt;?, ?&gt;&gt; currentState = dataTree.takeSnapshot().readNode(path);<br/> final DOMImmutableDataChangeEvent event;<br/> if (currentState.isPresent()) </p> { final NormalizedNode&lt;?, ?&gt; data = currentState.get(); event = DOMImmutableDataChangeEvent.builder(DataChangeScope.BASE).setAfter(data).addCreated(path, data).build(); } <p> else </p> { event = null; } <p>This is what we do when a dcn is registered in CDS. Notice that when we create the initial event we specify the scope as BASE. I am wondering if when this event is processed by the DataBroker it fails to match the event with the binding aware listeners and thus no event is generated on the binding side. I still need to confirm this but I suspect this may be the root cause.</p> <p>In my scenario, i register listener with the data change scope of SUBTREE. I can give it a quick try by changing the scope to BASE and see if it works.</p> <p>I changes the data change scope from SUBTREE to BASE for the listener registration, but no change in the behavior. I also tried with data change scope ONE and same results, No change.</p> <p>As I tested, I mimiced Anil's OVSDB registration by adding a "passenger" list to the car test model. The registration is:</p> <p>InstanceIdentifier&lt;Passenger&gt; path = InstanceIdentifier<br/> .create(Cars.class)<br/> .child(CarEntry.class, new CarEntryKey(new CarId("car1")))<br/> .child(Passenger.class);</p> <p>getDataBrokerDependency().registerDataChangeListener(<br/> LogicalDatastoreType.CONFIGURATION, path, new CarListener(),<br/> DataChangeScope.SUBTREE);</p> <p>So the last argument is wildcarded for the Passenger list.</p> <p>When I create a car with a passenger the listener gets notified as expected. But on restart it doesn't. The problem is that the read call:</p> <p> Optional&lt;NormalizedNode&lt;?, ?&gt;&gt; currentState = <br/> dataTree.takeSnapshot().readNode(path);</p> <p>returns absent. </p> <p>The DOM path is </p> <p> /(urn:sal-clustering-it:car?revision=2014-08-18)cars/car-entry<br/> /car-entry[</p> {(urn:sal-clustering-it:car?revision=2014-08-18)id=car1} <p>]<br/> /passenger/passenger</p> <p>The first path arg corresponds to the MapNode in the tree and the second represents the wildcarded MapNodeEntry instances. As NormalizedNodes.findNode traverses the path args, when it hits the last path arg, NormalizedNodes.getDirectChild expects a key path arg of type NodeIdentifierWithPredicates for a parent MapNode:</p> <p> else if (node instanceof MapNode &amp;&amp; pathArg instanceof <br/> NodeIdentifierWithPredicates) {<br/> return (Optional) ((MapNode) node)<br/> .getChild((NodeIdentifierWithPredicates) pathArg);</p> <p>However the actual path arg is NodeIdentifier because it's wildcarded.</p> <p>It seems the solution is to add a case for "node instanceof MapNode &amp;&amp; pathArg instanceof NodeIdentifier" to return the MapNode.</p> <p>In the comments below, I meant to say "The first "passenger" path arg corresponds to the MapNode ..."</p> <p>(In reply to Tom Pantelis from comment #11)<br/> &gt; As I tested, I mimiced Anil's OVSDB registration by adding a "passenger"<br/> &gt; list to the car test model. The registration is:<br/> &gt; <br/> &gt; InstanceIdentifier&lt;Passenger&gt; path = InstanceIdentifier<br/> &gt; .create(Cars.class)<br/> &gt; .child(CarEntry.class, new CarEntryKey(new CarId("car1")))<br/> &gt; .child(Passenger.class);<br/> &gt; <br/> &gt; getDataBrokerDependency().registerDataChangeListener(<br/> &gt; LogicalDatastoreType.CONFIGURATION, path, new CarListener(),<br/> &gt; DataChangeScope.SUBTREE);<br/> &gt; <br/> &gt; So the last argument is wildcarded for the Passenger list.<br/> &gt; <br/> &gt; When I create a car with a passenger the listener gets notified as expected.<br/> &gt; But on restart it doesn't. The problem is that the read call:<br/> &gt; <br/> &gt; Optional&lt;NormalizedNode&lt;?, ?&gt;&gt; currentState = <br/> &gt; dataTree.takeSnapshot().readNode(path);<br/> &gt; <br/> &gt; returns absent. <br/> &gt; <br/> &gt; The DOM path is <br/> &gt; <br/> &gt; /(urn:sal-clustering-it:car?revision=2014-08-18)cars/car-entry<br/> &gt; /car-entry[</p> {(urn:sal-clustering-it:car?revision=2014-08-18)id=car1} <p>]<br/> &gt; /passenger/passenger<br/> &gt; <br/> &gt; The first path arg corresponds to the MapNode in the tree and the second<br/> &gt; represents the wildcarded MapNodeEntry instances. As<br/> &gt; NormalizedNodes.findNode traverses the path args, when it hits the last path<br/> &gt; arg, NormalizedNodes.getDirectChild expects a key path arg of type<br/> &gt; NodeIdentifierWithPredicates for a parent MapNode:<br/> &gt; <br/> &gt; else if (node instanceof MapNode &amp;&amp; pathArg instanceof <br/> &gt; NodeIdentifierWithPredicates) {<br/> &gt; return (Optional) ((MapNode) node)<br/> &gt; .getChild((NodeIdentifierWithPredicates) pathArg);<br/> &gt; <br/> &gt; However the actual path arg is NodeIdentifier because it's wildcarded.<br/> &gt; <br/> &gt; It seems the solution is to add a case for "node instanceof MapNode &amp;&amp;<br/> &gt; pathArg instanceof NodeIdentifier" to return the MapNode.</p> <p>Adding the case as I mentioned previously might easily work if the wildcarded path arg is at the end but would be trickier if in the middle.</p> <p>The binding read interface does not allow wildcarded reads - I tried reading the same path as was used to register and it throw an IllegalArgumentEx. So naturally neither does the DOM API, although it returns absent instead of throwing an ex.</p> <p>Maybe the solution is to read the root and let the ResolveDataChangeEventsTask code sort it out. This may be expensive but we could offload it to the listener actor. </p> <p>Tony - what do you think?</p> <p>I would not read root, but rather up to first wildcard and then let ResolveDataChangeEvent to resolve this.</p> <p>Will do.</p> <p>(In reply to Tony Tkacik from comment #14)<br/> &gt; I would not read root, but rather up to first wildcard and then let<br/> &gt; ResolveDataChangeEvent to resolve this.</p> <p>stable/lithium patch <a href="https://git.opendaylight.org/gerrit/#/c/26548/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/26548/</a></p> <p>testing the fix..</p> <p>I tested the above patch and it works fine. I can see the Data Change blob when i restart the controller.</p> <p>I changed the state to Verified/Fixed, let me know if i need to change it back to Resolved/Fixed.</p> Duplicate CONTROLLER-1410 Development External issue ID 4094 External issue URL Issue Type ODL SR Target Milestone Rank 0|i02qbz: