While working on the new style end2end component ElanServiceTest in (ongoing, WIP) https://git.opendaylight.org/gerrit/#/c/44000/, I've just hit the exception shown below.

The test (currently local code, will push to c/44000 shortly) is really simple, it does a createElanInstance() on an IElanService in first @Test public void createElanInstance(), which works but the test then fails in the assert comparison due to another completely unrelated problem, so it quickly moves on to the next test, which does createElanInstance() again and then addElanInterface(), and that's what fails with below.

This code itself is simple single threaded - but the elanmanager datatree listeners are async, of course...

While I don't fully understand what's going on here yet, I suspect that perhaps one of the background threads processing from aysnc data store change listeners modified the data store concurrently and just before the Tx submit() in the addElanInterface() ?

Does the error below mean some sort of like optimistic lock violation? Are we missing some sort of retry logic? Do we have utilities for this? Is this a problem particular to 1 line in elanmanager, or a more general issue that we need to properly retry data store write actions everywhere?

java.lang.RuntimeException: TransactionCommitFailedException

{message=commit execution failed, errorList=[RpcError [message=commit execution failed, severity=ERROR, errorType=APPLICATION, tag=operation-failed, applicationTag=null, info=null, cause=java.lang.IllegalStateException: Store tree org.opendaylight.yangtools.yang.data.api.schema.tree.spi.MaterializedContainerNode@5243d730 and candidate base org.opendaylight.yangtools.yang.data.api.schema.tree.spi.MaterializedContainerNode@45b2d17b differ.]]}

at org.opendaylight.genius.mdsalutil.MDSALUtil.syncWrite(MDSALUtil.java:536)
at org.opendaylight.netvirt.elan.internal.ElanServiceProvider.addElanInterface(ElanServiceProvider.java:277)
at org.opendaylight.netvirt.elanmanager.tests.ElanServiceTest.addElanInterface(ElanServiceTest.java:80)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.opendaylight.infrautils.inject.guice.testutils.GuiceRule$1.evaluate(GuiceRule.java:83)
at org.opendaylight.infrautils.testutils.RunUntilFailureRule$RunUntilFailureStatement.evaluate(RunUntilFailureRule.java:59)
at org.opendaylight.infrautils.testutils.LogRule$1.evaluate(LogRule.java:45)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
at org.opendaylight.infrautils.testutils.RunUntilFailureClassRule$RunUntilFailureStatement.evaluate(RunUntilFailureClassRule.java:46)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:86)
at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:459)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:678)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:382)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:192)
Caused by: TransactionCommitFailedException

{message=commit execution failed, errorList=[RpcError [message=commit execution failed, severity=ERROR, errorType=APPLICATION, tag=operation-failed, applicationTag=null, info=null, cause=java.lang.IllegalStateException: Store tree org.opendaylight.yangtools.yang.data.api.schema.tree.spi.MaterializedContainerNode@5243d730 and candidate base org.opendaylight.yangtools.yang.data.api.schema.tree.spi.MaterializedContainerNode@45b2d17b differ.]]}

at org.opendaylight.controller.md.sal.dom.broker.impl.TransactionCommitFailedExceptionMapper.newWithCause(TransactionCommitFailedExceptionMapper.java:37)
at org.opendaylight.controller.md.sal.dom.broker.impl.TransactionCommitFailedExceptionMapper.newWithCause(TransactionCommitFailedExceptionMapper.java:18)
at org.opendaylight.yangtools.util.concurrent.ExceptionMapper.apply(ExceptionMapper.java:87)
at org.opendaylight.yangtools.util.concurrent.ExceptionMapper.apply(ExceptionMapper.java:37)
at org.opendaylight.yangtools.util.concurrent.MappingCheckedFuture.mapException(MappingCheckedFuture.java:60)
at org.opendaylight.yangtools.util.concurrent.MappingCheckedFuture.wrapInExecutionException(MappingCheckedFuture.java:64)
at org.opendaylight.yangtools.util.concurrent.MappingCheckedFuture.get(MappingCheckedFuture.java:77)
at com.google.common.util.concurrent.AbstractCheckedFuture.checkedGet(AbstractCheckedFuture.java:78)
at org.opendaylight.genius.datastoreutils.SingleTransactionDataBroker.syncWrite(SingleTransactionDataBroker.java:178)
at org.opendaylight.genius.mdsalutil.MDSALUtil.syncWrite(MDSALUtil.java:533)
... 32 more
Caused by: java.lang.IllegalStateException: Store tree org.opendaylight.yangtools.yang.data.api.schema.tree.spi.MaterializedContainerNode@5243d730 and candidate base org.opendaylight.yangtools.yang.data.api.schema.tree.spi.MaterializedContainerNode@45b2d17b differ.
at org.opendaylight.yangtools.yang.data.impl.schema.tree.InMemoryDataTree.commit(InMemoryDataTree.java:119)
at org.opendaylight.controller.md.sal.dom.store.impl.InMemoryDOMDataStore.commit(InMemoryDOMDataStore.java:248)
at org.opendaylight.controller.md.sal.dom.store.impl.InMemoryDOMStoreThreePhaseCommitCohort.commit(InMemoryDOMStoreThreePhaseCommitCohort.java:105)
at org.opendaylight.controller.md.sal.dom.broker.impl.CommitCoordinationTask.commitAll(CommitCoordinationTask.java:211)
at org.opendaylight.controller.md.sal.dom.broker.impl.CommitCoordinationTask.commitBlocking(CommitCoordinationTask.java:186)
at org.opendaylight.controller.md.sal.dom.broker.impl.CommitCoordinationTask.call(CommitCoordinationTask.java:65)
at org.opendaylight.controller.md.sal.dom.broker.impl.CommitCoordinationTask.call(CommitCoordinationTask.java:29)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at com.google.common.util.concurrent.MoreExecutors$DirectExecutorService.execute(MoreExecutors.java:299)
at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134)
at com.google.common.util.concurrent.AbstractListeningExecutorService.submit(AbstractListeningExecutorService.java:58)
at org.opendaylight.controller.md.sal.dom.broker.impl.SerializedDOMDataBroker.submit(SerializedDOMDataBroker.java:70)
at org.opendaylight.controller.md.sal.dom.broker.impl.DOMForwardedWriteTransaction.submit(DOMForwardedWriteTransaction.java:144)
at org.opendaylight.controller.md.sal.binding.impl.AbstractWriteTransaction.doSubmit(AbstractWriteTransaction.java:134)
at org.opendaylight.controller.md.sal.binding.impl.BindingDOMWriteTransactionAdapter.submit(BindingDOMWriteTransactionAdapter.java:83)
... 34 more