https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/YANGTOOLS-656 yangtools <p>Remote peer advertise 1M routes then it is disconnected, empty topology is expected, NPE appeared instead:</p> <p>araf | command | 306 - org.apache.karaf.log.command - 3.0.7 | ROBOT MESSAGE: Starting test Kill_Talking_BGP_Speaker<br/> 2016-08-30 02:41:19,795 | INFO | entLoopGroup-5-1 | BGPSessionImpl | 185 - org.opendaylight.bgpcep.bgp-rib-impl - 0.6.0.SNAPSHOT | End of input detected. Close the session.<br/> 2016-08-30 02:41:19,795 | INFO | entLoopGroup-5-1 | BGPPeer | 185 - org.opendaylight.bgpcep.bgp-rib-impl - 0.6.0.SNAPSHOT | Session with peer 10.29.12.104 went down<br/> 2016-08-30 02:41:19,809 | INFO | entLoopGroup-5-1 | BGPSessionImpl | 185 - org.opendaylight.bgpcep.bgp-rib-impl - 0.6.0.SNAPSHOT | Closing session: BGPSessionImpl</p> {channel=[id: 0x45b8e831, L:/10.29.13.40:1790 ! R:/10.29.12.104:17900], state=UP} <p>2016-08-30 02:41:19,919 | INFO | h for user karaf | command | 306 - org.apache.karaf.log.command - 3.0.7 | ROBOT MESSAGE: Starting test Store_Results_For_Talking_BGP_Speaker<br/> 2016-08-30 02:41:20,375 | INFO | h for user karaf | command | 306 - org.apache.karaf.log.command - 3.0.7 | ROBOT MESSAGE: Starting test Wait_For_Stable_Ipv4_Topology_After_Talking<br/> 2016-08-30 02:41:28,659 | WARN | on-dispatcher-46 | AbstractTopologyBuilder | 191 - org.opendaylight.bgpcep.bgp-topology-provider - 0.6.0.SNAPSHOT | Data change LazyDataTreeModification{path = DataTreeIdentifier{datastoreType = OPERATIONAL, rootIdentifier = KeyedInstanceIdentifier{targetType=interface org.opendaylight.yang.gen.v1.urn.opendaylight.params.xml.ns.yang.bgp.inet.rev150305.ipv4.routes.ipv4.routes.Ipv4Route, path=[org.opendaylight.yang.gen.v1.urn.opendaylight.params.xml.ns.yang.bgp.rib.rev130925.BgpRib, org.opendaylight.yang.gen.v1.urn.opendaylight.params.xml.ns.yang.bgp.rib.rev130925.bgp.rib.Rib[key=RibKey [_id=Uri <span class="error">&#91;_value=example-bgp-rib&#93;</span>]], org.opendaylight.yang.gen.v1.urn.opendaylight.params.xml.ns.yang.bgp.rib.rev130925.bgp.rib.rib.LocRib, org.opendaylight.yang.gen.v1.urn.opendaylight.params.xml.ns.yang.bgp.rib.rev130925.rib.Tables[key=TablesKey <span class="error">&#91;_afi=class org.opendaylight.yang.gen.v1.urn.opendaylight.params.xml.ns.yang.bgp.types.rev130919.Ipv4AddressFamily, _safi=class org.opendaylight.yang.gen.v1.urn.opendaylight.params.xml.ns.yang.bgp.types.rev130919.UnicastSubsequentAddressFamily&#93;</span>], org.opendaylight.yang.gen.v1.urn.opendaylight.params.xml.ns.yang.bgp.inet.rev150305.ipv4.routes.Ipv4Routes, org.opendaylight.yang.gen.v1.urn.opendaylight.params.xml.ns.yang.bgp.inet.rev150305.ipv4.routes.ipv4.routes.Ipv4Route[key=Ipv4RouteKey [_prefix=Ipv4Prefix <span class="error">&#91;_value=9.84.88.112/28&#93;</span>, _pathId=PathId <span class="error">&#91;_value=0&#93;</span>]]]}}, rootNode = LazyDataObjectModification{identifier = org.opendaylight.yang.gen.v1.urn.opendaylight.params.xml.ns.yang.bgp.inet.rev150305.ipv4.routes.ipv4.routes.Ipv4Route[key=Ipv4RouteKey [_prefix=Ipv4Prefix <span class="error">&#91;_value=9.84.88.112/28&#93;</span>, _pathId=PathId <span class="error">&#91;_value=0&#93;</span>]], domData = ChildNode{mod = NodeModification [identifier=(urn:opendaylight:params:xml:ns:yang:bgp-inet?revision=2015-03-05)ipv4-route[</p> {(urn:opendaylight:params:xml:ns:yang:bgp-inet?revision=2015-03-05)path-id=0, (urn:opendaylight:params:xml:ns:yang:bgp-inet?revision=2015-03-05)prefix=9.84.88.112/28} <p>], modificationType=DELETE, childModification={}], oldMeta = null, newMeta = null}}} (transaction DOM-CHAIN-4-364481) was not completely propagated to listener org.opendaylight.bgpcep.bgp.topology.provider.Ipv4ReachabilityTopologyBuilder@245a6da5<br/> java.lang.NullPointerException<br/> at org.opendaylight.bgpcep.bgp.topology.provider.Ipv4ReachabilityTopologyBuilder.getAttributes(Ipv4ReachabilityTopologyBuilder.java:41)<span class="error">&#91;191:org.opendaylight.bgpcep.bgp-topology-provider:0.6.0.SNAPSHOT&#93;</span><br/> at org.opendaylight.bgpcep.bgp.topology.provider.Ipv4ReachabilityTopologyBuilder.getAttributes(Ipv4ReachabilityTopologyBuilder.java:27)<span class="error">&#91;191:org.opendaylight.bgpcep.bgp-topology-provider:0.6.0.SNAPSHOT&#93;</span><br/> at org.opendaylight.bgpcep.bgp.topology.provider.AbstractReachabilityTopologyBuilder.removeObject(AbstractReachabilityTopologyBuilder.java:145)<span class="error">&#91;191:org.opendaylight.bgpcep.bgp-topology-provider:0.6.0.SNAPSHOT&#93;</span><br/> at org.opendaylight.bgpcep.bgp.topology.provider.AbstractTopologyBuilder.routeChanged(AbstractTopologyBuilder.java:213)<span class="error">&#91;191:org.opendaylight.bgpcep.bgp-topology-provider:0.6.0.SNAPSHOT&#93;</span><br/> at org.opendaylight.bgpcep.bgp.topology.provider.AbstractTopologyBuilder.onDataTreeChanged(AbstractTopologyBuilder.java:175)<span class="error">&#91;191:org.opendaylight.bgpcep.bgp-topology-provider:0.6.0.SNAPSHOT&#93;</span><br/> at org.opendaylight.controller.md.sal.binding.impl.BindingDOMDataTreeChangeListenerAdapter.onDataTreeChanged(BindingDOMDataTreeChangeListenerAdapter.java:41)<span class="error">&#91;143:org.opendaylight.controller.sal-binding-broker-impl:1.4.0.SNAPSHOT&#93;</span><br/> at org.opendaylight.controller.cluster.datastore.DataTreeChangeListenerActor.dataChanged(DataTreeChangeListenerActor.java:55)<span class="error">&#91;172:org.opendaylight.controller.sal-distributed-datastore:1.4.0.SNAPSHOT&#93;</span><br/> at org.opendaylight.controller.cluster.datastore.DataTreeChangeListenerActor.handleReceive(DataTreeChangeListenerActor.java:37)<span class="error">&#91;172:org.opendaylight.controller.sal-distributed-datastore:1.4.0.SNAPSHOT&#93;</span><br/> at org.opendaylight.controller.cluster.common.actor.AbstractUntypedActor.onReceive(AbstractUntypedActor.java:26)<span class="error">&#91;167:org.opendaylight.controller.sal-clustering-commons:1.4.0.SNAPSHOT&#93;</span><br/> at akka.actor.UntypedActor$$anonfun$receive$1.applyOrElse(UntypedActor.scala:165)<span class="error">&#91;155:com.typesafe.akka.actor:2.4.7&#93;</span><br/> at akka.actor.Actor$class.aroundReceive(Actor.scala:484)<span class="error">&#91;155:com.typesafe.akka.actor:2.4.7&#93;</span><br/> at akka.actor.UntypedActor.aroundReceive(UntypedActor.scala:95)<span class="error">&#91;155:com.typesafe.akka.actor:2.4.7&#93;</span><br/> at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)<span class="error">&#91;155:com.typesafe.akka.actor:2.4.7&#93;</span><br/> at akka.actor.ActorCell.invoke(ActorCell.scala:495)<span class="error">&#91;155:com.typesafe.akka.actor:2.4.7&#93;</span><br/> at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)<span class="error">&#91;155:com.typesafe.akka.actor:2.4.7&#93;</span><br/> at akka.dispatch.Mailbox.run(Mailbox.scala:224)<span class="error">&#91;155:com.typesafe.akka.actor:2.4.7&#93;</span><br/> at akka.dispatch.Mailbox.exec(Mailbox.scala:234)<span class="error">&#91;155:com.typesafe.akka.actor:2.4.7&#93;</span><br/> at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)<span class="error">&#91;151:org.scala-lang.scala-library:2.11.8.v20160304-115712-1706a37eb8&#93;</span><br/> at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)<span class="error">&#91;151:org.scala-lang.scala-library:2.11.8.v20160304-115712-1706a37eb8&#93;</span><br/> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)<span class="error">&#91;151:org.scala-lang.scala-library:2.11.8.v20160304-115712-1706a37eb8&#93;</span><br/> at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)<span class="error">&#91;151:org.scala-lang.scala-library:2.11.8.v20160304-115712-1706a37eb8&#93;</span></p> <p>Operating System: All<br/> Platform: All</p> YANGTOOLS-656

Occasionally, DTCL delete notification does not include "data before"

Bug Resolved Done Robert Varga Milos Fabian Tue, 30 Aug 2016 08:54:11 +0000 Sun, 10 Apr 2022 18:35:51 +0000 Wed, 23 Nov 2016 13:22:44 +0000 0 7 <p>Appeared in ingest-mixed Boron CSIT suite</p> <p>Similar problem observed when testing locally:<br/> 1. run ODL with configured app-peer<br/> 2. inject 100k prefixes<br/> 3. remove 100k prefixes (one-by-one)<br/> 4. unexpected behavior observed in LocRibWriter - calling #get() on empty optional - modification type DELETE, dataBefore absent<br/> The 6 prefixes left in loc-rib</p> <p>(In reply to Milos Fabian from comment #2)<br/> &gt; Similar problem observed when testing locally:<br/> &gt; 1. run ODL with configured app-peer<br/> &gt; 2. inject 100k prefixes<br/> &gt; 3. remove 100k prefixes (one-by-one)<br/> &gt; 4. unexpected behavior observed in LocRibWriter - calling #get() on empty<br/> &gt; optional - modification type DELETE, dataBefore absent<br/> &gt; The 6 prefixes left in loc-rib</p> <p>Happening with some very specific prefixes:</p> <p>2016-08-30 16:36:40,242 | INFO | on-dispatcher-96 | LocRibWriter | 288 - org.opendaylight.bgpcep.bgp-rib-impl - 0.7.0.SNAPSHOT | Data before missing for: (urn:opendaylight:params:xml:ns:yang:bgp-inet?revision=2015-03-05)ipv4-route[</p> {(urn:opendaylight:params:xml:ns:yang:bgp-inet?revision=2015-03-05)prefix=1.2.120.198/32} <p>], ChildNode{mod = NodeModification [identifier=(urn:opendaylight:params:xml:ns:yang:bgp-inet?revision=2015-03-05)ipv4-route[</p> {(urn:opendaylight:params:xml:ns:yang:bgp-inet?revision=2015-03-05)prefix=1.2.120.198/32, (urn:opendaylight:params:xml:ns:yang:bgp-inet?revision=2015-03-05)path-id=0} <p>], modificationType=DELETE, childModification={}], oldMeta = null, newMeta = null}<br/> 2016-08-30 16:36:40,245 | INFO | on-dispatcher-96 | LocRibWriter | 288 - org.opendaylight.bgpcep.bgp-rib-impl - 0.7.0.SNAPSHOT | Data before missing for: (urn:opendaylight:params:xml:ns:yang:bgp-inet?revision=2015-03-05)ipv4-route[</p> {(urn:opendaylight:params:xml:ns:yang:bgp-inet?revision=2015-03-05)prefix=1.2.121.198/32} <p>], ChildNode{mod = NodeModification [identifier=(urn:opendaylight:params:xml:ns:yang:bgp-inet?revision=2015-03-05)ipv4-route[</p> {(urn:opendaylight:params:xml:ns:yang:bgp-inet?revision=2015-03-05)prefix=1.2.121.198/32, (urn:opendaylight:params:xml:ns:yang:bgp-inet?revision=2015-03-05)path-id=0} <p>], modificationType=DELETE, childModification={}], oldMeta = null, newMeta = null}<br/> 2016-08-30 16:36:40,250 | INFO | on-dispatcher-96 | LocRibWriter | 288 - org.opendaylight.bgpcep.bgp-rib-impl - 0.7.0.SNAPSHOT | Data before missing for: (urn:opendaylight:params:xml:ns:yang:bgp-inet?revision=2015-03-05)ipv4-route[</p> {(urn:opendaylight:params:xml:ns:yang:bgp-inet?revision=2015-03-05)prefix=1.2.121.197/32} <p>], ChildNode{mod = NodeModification [identifier=(urn:opendaylight:params:xml:ns:yang:bgp-inet?revision=2015-03-05)ipv4-route[</p> {(urn:opendaylight:params:xml:ns:yang:bgp-inet?revision=2015-03-05)prefix=1.2.121.197/32, (urn:opendaylight:params:xml:ns:yang:bgp-inet?revision=2015-03-05)path-id=0} <p>], modificationType=DELETE, childModification={}], oldMeta = null, newMeta = null}<br/> 2016-08-30 16:36:40,253 | INFO | on-dispatcher-96 | LocRibWriter | 288 - org.opendaylight.bgpcep.bgp-rib-impl - 0.7.0.SNAPSHOT | Data before missing for: (urn:opendaylight:params:xml:ns:yang:bgp-inet?revision=2015-03-05)ipv4-route[</p> {(urn:opendaylight:params:xml:ns:yang:bgp-inet?revision=2015-03-05)prefix=1.2.120.199/32} <p>], ChildNode{mod = NodeModification [identifier=(urn:opendaylight:params:xml:ns:yang:bgp-inet?revision=2015-03-05)ipv4-route[</p> {(urn:opendaylight:params:xml:ns:yang:bgp-inet?revision=2015-03-05)prefix=1.2.120.199/32, (urn:opendaylight:params:xml:ns:yang:bgp-inet?revision=2015-03-05)path-id=0} <p>], modificationType=DELETE, childModification={}], oldMeta = null, newMeta = null}<br/> 2016-08-30 16:36:40,255 | INFO | on-dispatcher-96 | LocRibWriter | 288 - org.opendaylight.bgpcep.bgp-rib-impl - 0.7.0.SNAPSHOT | Data before missing for: (urn:opendaylight:params:xml:ns:yang:bgp-inet?revision=2015-03-05)ipv4-route[</p> {(urn:opendaylight:params:xml:ns:yang:bgp-inet?revision=2015-03-05)prefix=1.2.121.199/32} <p>], ChildNode{mod = NodeModification [identifier=(urn:opendaylight:params:xml:ns:yang:bgp-inet?revision=2015-03-05)ipv4-route[</p> {(urn:opendaylight:params:xml:ns:yang:bgp-inet?revision=2015-03-05)prefix=1.2.121.199/32, (urn:opendaylight:params:xml:ns:yang:bgp-inet?revision=2015-03-05)path-id=0} <p>], modificationType=DELETE, childModification={}], oldMeta = null, newMeta = null}<br/> 2016-08-30 16:36:40,274 | INFO | on-dispatcher-96 | LocRibWriter | 288 - org.opendaylight.bgpcep.bgp-rib-impl - 0.7.0.SNAPSHOT | Data before missing for: (urn:opendaylight:params:xml:ns:yang:bgp-inet?revision=2015-03-05)ipv4-route[</p> {(urn:opendaylight:params:xml:ns:yang:bgp-inet?revision=2015-03-05)prefix=1.2.120.197/32} <p>], ChildNode{mod = NodeModification [identifier=(urn:opendaylight:params:xml:ns:yang:bgp-inet?revision=2015-03-05)ipv4-route[</p> {(urn:opendaylight:params:xml:ns:yang:bgp-inet?revision=2015-03-05)prefix=1.2.120.197/32, (urn:opendaylight:params:xml:ns:yang:bgp-inet?revision=2015-03-05)path-id=0} <p>], modificationType=DELETE, childModification={}], oldMeta = null, newMeta = null}</p> <p>The problem addressed to controller/md-sal devs: <a href="https://lists.opendaylight.org/pipermail/controller-dev/2016-August/012561.html" class="external-link" target="_blank" rel="nofollow noopener">https://lists.opendaylight.org/pipermail/controller-dev/2016-August/012561.html</a></p> <p>Same issue is seen with Beryllium as well. Please refer exception encountered in read-life network environment under high load:</p> <p>2016-09-01 20:22:04,696 | WARN | lt-dispatcher-42 | AbstractTopologyBuilder | 329 - org.opendaylight.bgpcep.bgp-topology-provider - 0.5.0.Beryllium | Data change org.opendaylight.controller.md.sal.binding.impl.LazyDataTreeModification@963f998 (transaction DOM-CHAIN-10-159563) was not completely propagated to listener org.opendaylight.bgpcep.bgp.topology.provider.LinkstateTopologyBuilder@2976ef9<br/> java.lang.NullPointerException<br/> at org.opendaylight.bgpcep.bgp.topology.provider.UriBuilder.&lt;init&gt;(UriBuilder.java:43)<span class="error">&#91;329:org.opendaylight.bgpcep.bgp-topology-provider:0.5.0.Beryllium&#93;</span><br/> at org.opendaylight.bgpcep.bgp.topology.provider.LinkstateTopologyBuilder.removeObject(LinkstateTopologyBuilder.java:933)<span class="error">&#91;329:org.opendaylight.bgpcep.bgp-topology-provider:0.5.0.Beryllium&#93;</span><br/> at org.opendaylight.bgpcep.bgp.topology.provider.LinkstateTopologyBuilder.removeObject(LinkstateTopologyBuilder.java:124)<span class="error">&#91;329:org.opendaylight.bgpcep.bgp-topology-provider:0.5.0.Beryllium&#93;</span><br/> at org.opendaylight.bgpcep.bgp.topology.provider.AbstractTopologyBuilder.routeChanged(AbstractTopologyBuilder.java:213)<span class="error">&#91;329:org.opendaylight.bgpcep.bgp-topology-provider:0.5.0.Beryllium&#93;</span><br/> at org.opendaylight.bgpcep.bgp.topology.provider.AbstractTopologyBuilder.onDataTreeChanged(AbstractTopologyBuilder.java:175)<span class="error">&#91;329:org.opendaylight.bgpcep.bgp-topology-provider:0.5.0.Beryllium&#93;</span><br/> at org.opendaylight.controller.md.sal.binding.impl.BindingDOMDataTreeChangeListenerAdapter.onDataTreeChanged(BindingDOMDataTreeChangeListenerAdapter.java:41)<span class="error">&#91;115:org.opendaylight.controller.sal-binding-broker-impl:1.3.0.Beryllium&#93;</span><br/> at org.opendaylight.controller.cluster.datastore.DataTreeChangeListenerActor.dataChanged(DataTreeChangeListenerActor.java:53)<span class="error">&#91;140:org.opendaylight.controller.sal-distributed-datastore:1.3.0.Beryllium&#93;</span><br/> at org.opendaylight.controller.cluster.datastore.DataTreeChangeListenerActor.handleReceive(DataTreeChangeListenerActor.java:37)<span class="error">&#91;140:org.opendaylight.controller.sal-distributed-datastore:1.3.0.Beryllium&#93;</span><br/> at org.opendaylight.controller.cluster.common.actor.AbstractUntypedActor.onReceive(AbstractUntypedActor.java:34)<span class="error">&#91;136:org.opendaylight.controller.sal-clustering-commons:1.3.0.Beryllium&#93;</span></p> <p>Can you report the entire DTCN, from root for this case?</p> <p>This looks like some badness in the DataTree implementation. Specifically it looks like AbstractModifiedNodeBasedCandidateNode is allocating a ChildNode with no backing metadata.</p> <p>Now, that child node was resolved to DELETE (in SchemaAwareApplyOperation.apply()), which indicates that the 'before' metadata was supplied to it, while it was not actually present in the tree. This evidence points it squarely at StructuralContainerModificationStrategy#57, which is the only place playing this kind of a trick.</p> <p>We need logs replicating this issue, with <a href="https://git.opendaylight.org/gerrit/48049" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/48049</a> applied to the controller and ShardDataTree logging set to TRACE.</p> <p>We need to see both the modification and the data tree candidate which trigger this behavior.</p> <p>Attachment karaf.log.tar.gz has been added with description: log</p> <p>Attachment karaf.log.1.tar.gz has been added with description: log1</p> <p>Attachment karaf.log.2.tar.gz has been added with description: log2</p> <p>Attachment karaf.log.3.tar.gz has been added with description: log3</p> <p>Attachment karaf.log.4.tar.gz has been added with description: log4</p> <p>Analyzing karaf.log.4, the problem is narrowed down a bit. In this specific case, the route being deleted is: prefix=1.2.120.199/32 path-id=0</p> <p>The deletion record is correct, as the route is present in parent's oldMeta and it is not present in its newMeta.</p> <p>The stack trace has a Iterators$8.transform() as the caller of AbstractModifiedNodeBasedCandidateNode.childNode(), which points to its getChildNodes() method, which is logical, as the parent is marked as SUBTREE_MODIFIED and BGP code is traversing it to find out what is going on.</p> <p>Based on this, the cause seems to be the fact that eventhough the record is in oldMeta's children map, the call AbstractModifiedNodeBasedCandidateNode.childMeta() -&gt; MaterializedContainerNode.getChild() fails to find it &#8211; but it is registering it in its children map.</p> <p>Since the objects in question are supposed to be immutable at this point, I think it is safe to say that the DataTree code itself is correct and the problem lies in one of three components:</p> <ul class="alternate" type="square"> <li>NodeIdentifierWithPredicates</li> <li>TrieMap (since we are at a scale where children are tracked using it)</li> <li>JVM badness (affecting either one of the above)</li> </ul> <p>Actually, there is a fourth component involved: ImmutableOffsetMap, which is the backing map inside NodeIdentifierWithPredicates.</p> <p>Ajay, can you check what was the JVM version involved in that, please?</p> <p>debug patch to eliminate custom maps from the picture: <a href="https://git.opendaylight.org/gerrit/48420" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/48420</a></p> <p>tested against u112 and ImmutableMaps, issue still present</p> <p>Attachment BUG_6577_STEPS.txt has been added with description: Steps to reproduce bug</p> <p>After additional testing it seems the problem is either in JVM or TrieMap. Specifially, I have added code, which when it encounters this conditions walks through the map's entrySet() and tries to find the corresponding key.</p> <p>What I am getting is:</p> <p>2016-11-18 18:00:17,821 | WARN | lt-dispatcher-36 | ctModifiedNodeBasedCandidateNode | 85 - org.opendaylight.yangtools.yang-data-impl - 1.0.2.SNAPSHOT | Missing key is (urn:opendaylight:params:xml:ns:yang:bgp-inet?revision=2015-03-05)ipv4-route[</p> {(urn:opendaylight:params:xml:ns:yang:bgp-inet?revision=2015-03-05)path-id=0, (urn:opendaylight:params:xml:ns:yang:bgp-inet?revision=2015-03-05)prefix=1.2.121.197/32} <p>], identity 1745067963, hash 1674416874<br/> 2016-11-18 18:00:17,825 | WARN | lt-dispatcher-36 | ctModifiedNodeBasedCandidateNode | 85 - org.opendaylight.yangtools.yang-data-impl - 1.0.2.SNAPSHOT | Found matching key (urn:opendaylight:params:xml:ns:yang:bgp-inet?revision=2015-03-05)ipv4-route[</p> {(urn:opendaylight:params:xml:ns:yang:bgp-inet?revision=2015-03-05)path-id=0, (urn:opendaylight:params:xml:ns:yang:bgp-inet?revision=2015-03-05)prefix=1.2.121.197/32} <p>], identity 1269043004, hash 1674416874<br/> 2016-11-18 18:00:17,825 | WARN | lt-dispatcher-36 | ctModifiedNodeBasedCandidateNode | 85 - org.opendaylight.yangtools.yang-data-impl - 1.0.2.SNAPSHOT | Inverse equals: true<br/> 2016-11-18 18:00:17,826 | WARN | lt-dispatcher-36 | ctModifiedNodeBasedCandidateNode | 85 - org.opendaylight.yangtools.yang-data-impl - 1.0.2.SNAPSHOT | Stored lookup: SimpleContainerNode{version=org.opendaylight.yangtools.yang.data.api.schema.tree.spi.Version@267ba74b, data=ImmutableMapEntryNode{nodeIdentifier=(urn:opendaylight:params:xml:ns:yang:bgp-inet?revision=2015-03-05)ipv4-route[</p> {(urn:opendaylight:params:xml:ns:yang:bgp-inet?revision=2015-03-05)path-id=0, (urn:opendaylight:params:xml:ns:yang:bgp-inet?revision=2015-03-05)prefix=1.2.121.197/32} <p>], value=[ImmutableLeafNode{nodeIdentifier=(urn:opendaylight:params:xml:ns:yang:bgp-inet?revision=2015-03-05)path-id, value=0, attributes={}}, ImmutableLeafNode{nodeIdentifier=(urn:opendaylight:params:xml:ns:yang:bgp-inet?revision=2015-03-05)prefix, value=1.2.121.197/32, attributes={}}, ImmutableContainerNode{nodeIdentifier=(urn:opendaylight:params:xml:ns:yang:bgp-inet?revision=2015-03-05)attributes, value=[ImmutableContainerNode{nodeIdentifier=(urn:opendaylight:params:xml:ns:yang:bgp-inet?revision=2015-03-05)as-path, value=[], attributes={}}, ImmutableContainerNode{nodeIdentifier=(urn:opendaylight:params:xml:ns:yang:bgp-inet?revision=2015-03-05)origin, value=<span class="error">&#91;ImmutableLeafNode{nodeIdentifier=(urn:opendaylight:params:xml:ns:yang:bgp-inet?revision=2015-03-05)value, value=igp, attributes={}}&#93;</span>, attributes={}}, ImmutableContainerNode{nodeIdentifier=(urn:opendaylight:params:xml:ns:yang:bgp-inet?revision=2015-03-05)local-pref, value=<span class="error">&#91;ImmutableLeafNode{nodeIdentifier=(urn:opendaylight:params:xml:ns:yang:bgp-inet?revision=2015-03-05)pref, value=100, attributes={}}&#93;</span>, attributes={}}, ImmutableChoiceNode{nodeIdentifier=(urn:opendaylight:params:xml:ns:yang:bgp-inet?revision=2015-03-05)c-next-hop, value=[ImmutableContainerNode{nodeIdentifier=(urn:opendaylight:params:xml:ns:yang:bgp-inet?revision=2015-03-05)ipv4-next-hop, value=<span class="error">&#91;ImmutableLeafNode{nodeIdentifier=(urn:opendaylight:params:xml:ns:yang:bgp-inet?revision=2015-03-05)global, value=127.0.0.1, attributes={}}&#93;</span>, attributes={}}]}, ImmutableContainerNode{nodeIdentifier=(urn:opendaylight:params:xml:ns:yang:bgp-inet?revision=2015-03-05)multi-exit-disc, value=<span class="error">&#91;ImmutableLeafNode{nodeIdentifier=(urn:opendaylight:params:xml:ns:yang:bgp-inet?revision=2015-03-05)med, value=0, attributes={}}&#93;</span>, attributes={}}], attributes={}}], attributes={}}}<br/> 2016-11-18 18:00:17,826 | WARN | lt-dispatcher-36 | ctModifiedNodeBasedCandidateNode | 85 - org.opendaylight.yangtools.yang-data-impl - 1.0.2.SNAPSHOT | Normal lookup: null</p> <p>So the key is present as a different object in the map, they compare as .equals() in both directions and report the same hash code.</p> <p>When we perform the lookup using the stored key, the map returns a value. When we perform the lookup using our key, the map returns null &#8211; which is a violation of java.util.Map contract.</p> <p>Either the key's identity hash or identity comparison is used somewhere. Looking through TrieMap implementation, I do not see identity hash being used, nor keys being compared in any other way than through .equals().</p> <p>Next step will be to retry with JDK 1.8.0u60, to see if this problem is present.</p> <p>Tracked down to <a href="https://github.com/romix/java-concurrent-hash-trie-map/blob/master/src/main/java/com/romix/scala/collection/concurrent/TrieMap.java#L464" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/romix/java-concurrent-hash-trie-map/blob/master/src/main/java/com/romix/scala/collection/concurrent/TrieMap.java#L464</a>, which looks like a bug. The same code is present in the Scala version of the code at <a href="https://github.com/scala/scala/blob/2.12.x/src/library/scala/collection/concurrent/TrieMap.scala#L264" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/scala/scala/blob/2.12.x/src/library/scala/collection/concurrent/TrieMap.scala#L264</a>.</p> <p>I am not sure why identity comparison is used herre specifically, and the codepath only ever triggers for read-only trie &#8211; hence I guess it is not as well-tested as the read-write case.</p> <p>upstream: <a href="https://github.com/romix/java-concurrent-hash-trie-map/pull/24" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/romix/java-concurrent-hash-trie-map/pull/24</a></p> <p>scala: <a href="https://github.com/scala/scala/pull/5547" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/scala/scala/pull/5547</a></p> <p>Waiting for either an upstream release or helpdesk finish ticket #33071.</p> <p>ODL-private package seeded.</p> <p>master: <a href="https://git.opendaylight.org/gerrit/48531" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/48531</a></p> <p>boron: <a href="https://git.opendaylight.org/gerrit/48611" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/48611</a></p> Development External References bgpcep_bgp_ingest_mixed_txt_singlepeer_changecount_check_listening_ipv4_topology_count <br/> bgpcep_bgp_ingest_mixed_txt_singlepeer_changecount_check_for_empty_ipv4_topology_after_talking External issue ID 6577 External issue URL Issue Type ODL SR Target Milestone Rank 0|i028kn: