[YANGTOOLS-1537] CME in AbstractNodeContainerModificationStrategy.checkChildPreconditions() Created: 04/Sep/23  Updated: 06/Sep/23  Resolved: 06/Sep/23

Status: Resolved
Project: yangtools
Component/s: data-impl
Affects Version/s: 4.0.15, 5.0.10, 6.0.12, 7.0.18, 8.0.10, 9.0.8, 11.0.1, 10.0.9
Fix Version/s: 9.0.9, 10.0.10, 11.0.2

Type: Bug Priority: Medium
Reporter: Robert Varga Assignee: Robert Varga
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Blocks
blocks YANGTOOLS-1539 Improve InMemoryDataTreeModification ... Resolved
Issue split
split from NETCONF-1146 Operational data of netconf-topology ... Resolved
Relates
relates to YANGTOOLS-1025 DataTreeModification does not require... Resolved

 Description   

We have observed the following:

2023-08-31T22:31:51,428 | WARN  | opendaylight-cluster-data-shard-dispatcher-29 | ShardDataTree                    | 210 - org.opendaylight.controller.sal-distributed-datastore - 8.0.1 | member-1-shard-topology-operational: Unexpected failure in validation phase
java.util.ConcurrentModificationException: null
	at java.util.HashMap$HashIterator.nextNode(HashMap.java:1597) ~[?:?]
	at java.util.HashMap$ValueIterator.next(HashMap.java:1625) ~[?:?]
	at org.opendaylight.yangtools.yang.data.tree.impl.AbstractNodeContainerModificationStrategy.checkChildPreconditions(AbstractNodeContainerModificationStrategy.java:409) ~[bundleFile:?]
	at org.opendaylight.yangtools.yang.data.tree.impl.AbstractNodeContainerModificationStrategy.checkMergeApplicable(AbstractNodeContainerModificationStrategy.java:396) ~[bundleFile:?]
	at org.opendaylight.yangtools.yang.data.tree.impl.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:111) ~[bundleFile:?]
	at org.opendaylight.yangtools.yang.data.tree.impl.AbstractNodeContainerModificationStrategy.checkChildPreconditions(AbstractNodeContainerModificationStrategy.java:415) ~[bundleFile:?]
	at org.opendaylight.yangtools.yang.data.tree.impl.AbstractNodeContainerModificationStrategy.checkMergeApplicable(AbstractNodeContainerModificationStrategy.java:396) ~[bundleFile:?]
	at org.opendaylight.yangtools.yang.data.tree.impl.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:111) ~[bundleFile:?]
	at org.opendaylight.yangtools.yang.data.tree.impl.AbstractNodeContainerModificationStrategy.checkChildPreconditions(AbstractNodeContainerModificationStrategy.java:415) ~[bundleFile:?]
	at org.opendaylight.yangtools.yang.data.tree.impl.AbstractNodeContainerModificationStrategy.checkMergeApplicable(AbstractNodeContainerModificationStrategy.java:396) ~[bundleFile:?]
	at org.opendaylight.yangtools.yang.data.tree.impl.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:111) ~[bundleFile:?]
	at org.opendaylight.yangtools.yang.data.tree.impl.AbstractNodeContainerModificationStrategy.checkChildPreconditions(AbstractNodeContainerModificationStrategy.java:415) ~[bundleFile:?]
	at org.opendaylight.yangtools.yang.data.tree.impl.AbstractNodeContainerModificationStrategy.checkMergeApplicable(AbstractNodeContainerModificationStrategy.java:396) ~[bundleFile:?]
	at org.opendaylight.yangtools.yang.data.tree.impl.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:111) ~[bundleFile:?]
	at org.opendaylight.yangtools.yang.data.tree.impl.AbstractNodeContainerModificationStrategy.checkChildPreconditions(AbstractNodeContainerModificationStrategy.java:415) ~[bundleFile:?]
	at org.opendaylight.yangtools.yang.data.tree.impl.AbstractNodeContainerModificationStrategy.checkMergeApplicable(AbstractNodeContainerModificationStrategy.java:396) ~[bundleFile:?]
	at org.opendaylight.yangtools.yang.data.tree.impl.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:111) ~[bundleFile:?]
	at org.opendaylight.yangtools.yang.data.tree.impl.AbstractNodeContainerModificationStrategy.checkChildPreconditions(AbstractNodeContainerModificationStrategy.java:415) ~[bundleFile:?]
	at org.opendaylight.yangtools.yang.data.tree.impl.AbstractNodeContainerModificationStrategy.checkTouchApplicable(AbstractNodeContainerModificationStrategy.java:374) ~[bundleFile:?]
	at org.opendaylight.yangtools.yang.data.tree.impl.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:109) ~[bundleFile:?]
	at org.opendaylight.yangtools.yang.data.tree.impl.AbstractDataTreeTip.validate(AbstractDataTreeTip.java:37) ~[bundleFile:?]


 Comments   
Comment by Robert Varga [ 04/Sep/23 ]

I have not found a codepath which would modify the maps in this codepath, but we have cross-thread interactions here. What I think is happening is that:

  1. since YANGTOOLS-1025, the sequencing in AbstractDataTreeTip.validate() does not do a full volatile read, only an acquire fence (via InMemoryDataTreeModification.isSealed())
  2. HashMap's modCount field is using plain stores
  3. InMemoryDataTreeModification.ready() issues only a full volatile fence at its beginning
  4. therefore all modCount end up not being visible after ready() completes
  5. therefore the iterator does NOT see them when it starts, but at some point during execution they do filter in, causing the CME

At the end of the day, what is saving us most of the time is:

  1. the fact that weird modifications do not usually happen
  2. the fact that users usually synchronize datastore access themselves, which acts as a fence
Generated at Wed Feb 07 20:56:26 UTC 2024 using Jira 8.20.10#820010-sha1:ace47f9899e9ee25d7157d59aa17ab06aee30d3d.