OpenDaylight JIRA https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 [NETVIRT-985] java.lang.OutOfMemoryError: Java heap space https://jira.opendaylight.org/browse/NETVIRT-985 netvirt <p>During perf/scale testing, we see OOM on ODL when a large number of neutron resources are created and deleted we see that ODL is killed due to OOM. Looking at the stdout when the JVM crashed due to OOM, we see:</p> <p>Heap dump file created <span class="error">&#91;3089813876 bytes in 19.232 secs&#93;</span><br/> Uncaught error from thread <span class="error">&#91;opendaylight-cluster-data-shard-dispatcher-144&#93;</span> shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem<span class="error">&#91;opendaylight-cluster-data&#93;</span><br/> java.lang.OutOfMemoryError: Java heap space<br/> at com.google.common.collect.RegularImmutableMap.createEntryArray(RegularImmutableMap.java:148)<br/> at com.google.common.collect.RegularImmutableMap.&lt;init&gt;(RegularImmutableMap.java:81)<br/> at com.google.common.collect.ImmutableMap.copyOf(ImmutableMap.java:294)<br/> at org.opendaylight.controller.cluster.datastore.persisted.FrontendHistoryMetadata.&lt;init&gt;(FrontendHistoryMetadata.java:40)<br/> at org.opendaylight.controller.cluster.datastore.FrontendHistoryMetadataBuilder.build(FrontendHistoryMetadataBuilder.java:54)<br/> at org.opendaylight.controller.cluster.datastore.FrontendClientMetadataBuilder$$Lambda$431/741495460.apply(Unknown Source)<br/> at com.google.common.collect.Iterators$8.transform(Iterators.java:799)<br/> at com.google.common.collect.TransformedIterator.next(TransformedIterator.java:48)<br/> at java.util.AbstractCollection.toArray(AbstractCollection.java:141)<br/> at com.google.common.collect.ImmutableList.copyOf(ImmutableList.java:258)<br/> at org.opendaylight.controller.cluster.datastore.persisted.FrontendClientMetadata.&lt;init&gt;(FrontendClientMetadata.java:38)<br/> at org.opendaylight.controller.cluster.datastore.FrontendClientMetadataBuilder.build(FrontendClientMetadataBuilder.java:77)<br/> at org.opendaylight.controller.cluster.datastore.FrontendMetadata$$Lambda$430/2026307982.apply(Unknown Source)<br/> at com.google.common.collect.Iterators$8.transform(Iterators.java:799)<br/> at com.google.common.collect.TransformedIterator.next(TransformedIterator.java:48)<br/> at java.util.AbstractCollection.toArray(AbstractCollection.java:141)<br/> at com.google.common.collect.ImmutableList.copyOf(ImmutableList.java:258)<br/> at org.opendaylight.controller.cluster.datastore.persisted.FrontendShardDataTreeSnapshotMetadata.&lt;init&gt;(FrontendShardDataTreeSnapshotMetadata.java:71)<br/> at org.opendaylight.controller.cluster.datastore.FrontendMetadata.toSnapshot(FrontendMetadata.java:72)<br/> at org.opendaylight.controller.cluster.datastore.FrontendMetadata.toSnapshot(FrontendMetadata.java:33)<br/> at org.opendaylight.controller.cluster.datastore.ShardDataTree.takeStateSnapshot(ShardDataTree.java:216)<br/> at org.opendaylight.controller.cluster.datastore.ShardSnapshotCohort.createSnapshot(ShardSnapshotCohort.java:68)<br/> at org.opendaylight.controller.cluster.raft.RaftActorSnapshotMessageSupport.lambda$new$0(RaftActorSnapshotMessageSupport.java:52)<br/> at org.opendaylight.controller.cluster.raft.RaftActorSnapshotMessageSupport$$Lambda$123/1533883683.accept(Unknown Source)<br/> at org.opendaylight.controller.cluster.raft.SnapshotManager$Idle.capture(SnapshotManager.java:295)<br/> at org.opendaylight.controller.cluster.raft.SnapshotManager$Idle.capture(SnapshotManager.java:307)<br/> at org.opendaylight.controller.cluster.raft.SnapshotManager.capture(SnapshotManager.java:91)<br/> at org.opendaylight.controller.cluster.raft.behaviors.Follower.lambda$handleAppendEntries$0(Follower.java:254)<br/> at org.opendaylight.controller.cluster.raft.behaviors.Follower$$Lambda$127/742332312.apply(Unknown Source)<br/> at org.opendaylight.controller.cluster.raft.ReplicatedLogImpl.lambda$appendAndPersist$0(ReplicatedLogImpl.java:111)<br/> at org.opendaylight.controller.cluster.raft.ReplicatedLogImpl$$Lambda$128/559701765.apply(Unknown Source)<br/> at akka.persistence.UntypedPersistentActor$$anonfun$persist$1.apply(PersistentActor.scala:206)</p> NETVIRT-985 java.lang.OutOfMemoryError: Java heap space Bug Medium Resolved Done Michael Vorburger Sai Sindhur Malleni Fri, 10 Nov 2017 18:43:51 +0000 Tue, 8 May 2018 12:59:50 +0000 Thu, 30 Nov 2017 12:44:20 +0000 Carbon Nitrogen Carbon Oxygen 0 2 <p>Attached java_pid1_Leak_Suspects.zip from <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=smalleni" class="user-hover" rel="smalleni">smalleni</a> environment shows this another one like <a href="https://jira.opendaylight.org/browse/CONTROLLER-1756" title="OOM due to huge Map in ShardDataTree" class="issue-link" data-issue-key="CONTROLLER-1756"><del>CONTROLLER-1756</del></a> ... <img class="emoticon" src="https://jira.opendaylight.org/images/icons/emoticons/sad.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>Attached is the usual MAT produced Leak Suspects report, from a HPROF from 1st of 3 cluster nodes; the 2nd one show basically the same - 800 MB on 1 GB of total 2 GB due to huge Map in ShardDataTree.</p> <p>PS: The 3rd node does an OOM with a slightly different HPROF, see attached controller-2__java_pid1_Leak_Suspects.zip, showing 745 MB used by the thread MonitorableThreadFactory$AkkaForkJoinWorkerThread local variables, and 310 MB in org.opendaylight.controller.cluster.datastore.FrontendMetadata. I don't know enough about clustering internals to judge if that is just an impact of above, but unless educated otherwise, will have to suspect it is so.</p> <p>The Karaf log on controller-0 and controller-1 (both with the ShardDataTree OOM) do not contain "IllegalStateException: Attempted to close chain with outstanding transaction" as in <a href="https://jira.opendaylight.org/browse/CONTROLLER-1756" title="OOM due to huge Map in ShardDataTree" class="issue-link" data-issue-key="CONTROLLER-1756"><del>CONTROLLER-1756</del></a>, but a lot of other errors - too many to easily make any sense of them, so far... <img class="emoticon" src="https://jira.opendaylight.org/images/icons/emoticons/sad.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>Michael, shouldn't you be looking at logs on controller-1 and controller-2 since that is where the ODLs hit OOM.</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=smalleni" class="user-hover" rel="smalleni">smalleni</a> yeah you're right, I'll pour more through controller-1 log... but FYI you're controller-0 WOULD HAVE (was about to) OOM as well anyway... <img class="emoticon" src="https://jira.opendaylight.org/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>Grep'ing for "chain" in the attached karaf.log.xz from controller-1 reveals a lot of the following kinds messages again from openflowplugin which I have a hunch may have something to do with this:</p> <p>1. <em>"Transaction chain failed, recreating chain due to ..." :</em></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>2017-11-08 20:26:13,390 | WARN | CommitFutures-1 | TransactionChainManager | 304 - org.opendaylight.openflowplugin.common - 0.4.2.Carbon-redhat-2 | Transaction chain failed, recreating chain due to OptimisticLockFailedException{message=Optimistic lock failed., errorList=[RpcError [message=Optimistic lock failed., severity=ERROR, errorType=APPLICATION, tag=resource-denied, applicationTag=null, info=null, cause=org.opendaylight.yangtools.yang.data.api.schema.tree.ConflictingModificationAppliedException: Node was created by other transaction.]]} at org.opendaylight.controller.cluster.datastore.ShardDataTree.lambda$processNextPendingTransaction$0(ShardDataTree.java:751) at org.opendaylight.controller.cluster.datastore.ShardDataTree.processNextPending(ShardDataTree.java:789) at org.opendaylight.controller.cluster.datastore.ShardDataTree.processNextPendingTransaction(ShardDataTree.java:736) at org.opendaylight.controller.cluster.datastore.ShardDataTree.startCanCommit(ShardDataTree.java:819) at org.opendaylight.controller.cluster.datastore.SimpleShardDataTreeCohort.canCommit(SimpleShardDataTreeCohort.java:90) at org.opendaylight.controller.cluster.datastore.ChainedCommitCohort.canCommit(ChainedCommitCohort.java:58) (...) Caused by: org.opendaylight.yangtools.yang.data.api.schema.tree.ConflictingModificationAppliedException: Node was created by other transaction. at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkWriteApplicable(SchemaAwareApplyOperation.java:174) at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:128) at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkChildPreconditions(AbstractNodeContainerModificationStrategy.java:305) (...) </pre> </div></div> <p>2. <em>"WARN: ConflictingModificationAppliedException" &amp;&amp; "ERROR Exception during INITIAL transaction submitting. OptimisticLockFailedException" :</em></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>2017-11-09 13:28:33,002 | WARN | lt-dispatcher-41 | ConcurrentDOMDataBroker | 211 - org.opendaylight.controller.sal-distributed-datastore - 1.5.2.Carbon-redhat-2 | Tx: DOM-CHAIN-20-0 Error during phase CAN_COMMIT, starting Abort OptimisticLockFailedException{message=Optimistic lock failed., errorList=[RpcError [message=Optimistic lock failed., severity=ERROR, errorType=APPLICATION, tag=resource-denied, applicationTag=null, info=null, cause=org.opendaylight.yangtools.yang.data.ap i.schema.tree.ConflictingModificationAppliedException: Node children was modified by other transaction]]} at org.opendaylight.controller.cluster.datastore.ShardDataTree.lambda$processNextPendingTransaction$0(ShardDataTree.java:751) at org.opendaylight.controller.cluster.datastore.ShardDataTree.processNextPending(ShardDataTree.java:789) at org.opendaylight.controller.cluster.datastore.ShardDataTree.processNextPendingTransaction(ShardDataTree.java:736) at org.opendaylight.controller.cluster.datastore.ShardDataTree.startCanCommit(ShardDataTree.java:819) at org.opendaylight.controller.cluster.datastore.SimpleShardDataTreeCohort.canCommit(SimpleShardDataTreeCohort.java:90) at org.opendaylight.controller.cluster.datastore.ChainedCommitCohort.canCommit(ChainedCommitCohort.java:58) at org.opendaylight.controller.cluster.datastore.CohortEntry.canCommit(CohortEntry.java:97) at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator.handleCanCommit(ShardCommitCoordinator.java:236) at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator.handleForwardedReadyTransaction(ShardCommitCoordinator.java:114) at org.opendaylight.controller.cluster.datastore.Shard.handleForwardedReadyTransaction(Shard.java:699) at org.opendaylight.controller.cluster.datastore.Shard.handleNonRaftCommand(Shard.java:314) at org.opendaylight.controller.cluster.raft.RaftActor.handleCommand(RaftActor.java:270) at org.opendaylight.controller.cluster.common.actor.AbstractUntypedPersistentActor.onReceiveCommand(AbstractUntypedPersistentActor.java:44) at akka.persistence.UntypedPersistentActor.onReceive(PersistentActor.scala:170) at org.opendaylight.controller.cluster.common.actor.MeteringBehavior.apply(MeteringBehavior.java:104) at akka.actor.ActorCell$$anonfun$become$1.applyOrElse(ActorCell.scala:544) at akka.actor.Actor$class.aroundReceive(Actor.scala:502) at akka.persistence.UntypedPersistentActor.akka$persistence$Eventsourced$$super$aroundReceive(PersistentActor.scala:168) at akka.persistence.Eventsourced$$anon$1.stateReceive(Eventsourced.scala:727) at akka.persistence.Eventsourced$class.aroundReceive(Eventsourced.scala:183) at akka.persistence.UntypedPersistentActor.aroundReceive(PersistentActor.scala:168) at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526) at akka.actor.ActorCell.invoke(ActorCell.scala:495) at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257) at akka.dispatch.Mailbox.run(Mailbox.scala:224) at akka.dispatch.Mailbox.exec(Mailbox.scala:234) at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) Caused by: org.opendaylight.yangtools.yang.data.api.schema.tree.ConflictingModificationAppliedException: Node children was modified by other transaction at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkConflicting(SchemaAwareApplyOperation.java:78) at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkNotConflicting(SchemaAwareApplyOperation.java:108) at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkWriteApplicable(SchemaAwareApplyOperation.java:170) at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:128) at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkChildPreconditions(AbstractNodeContainerModificationStrategy.java:305) at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkMergeApplicable(AbstractNodeContainerModificationStrategy.java:313) at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:131) at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkChildPreconditions(AbstractNodeContainerModificationStrategy.java:305) at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkTouchApplicable(AbstractNodeContainerModificationStrategy.java:288) at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:125) at org.opendaylight.yangtools.yang.data.impl.schema.tree.StructuralContainerModificationStrategy.checkApplicable(StructuralContainerModificationStrategy.java:99) at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkChildPreconditions(AbstractNodeContainerModificationStrategy.java:305) at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkTouchApplicable(AbstractNodeContainerModificationStrategy.java:288) at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:125) at org.opendaylight.yangtools.yang.data.impl.schema.tree.RootModificationApplyOperation.checkApplicable(RootModificationApplyOperation.java:72) at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractDataTreeTip.validate(AbstractDataTreeTip.java:35) at org.opendaylight.controller.cluster.datastore.ShardDataTree.lambda$processNextPendingTransaction$0(ShardDataTree.java:743) ... 29 more 2017-11-09 13:28:33,003 | ERROR | ntLoopGroup-7-36 | TransactionChainManager | 304 - org.opendaylight.openflowplugin.common - 0.4.2.Carbon-redhat-2 | Exception during INITIAL transaction submitting. java.util.concurrent.ExecutionException: OptimisticLockFailedException{message=Optimistic lock failed., errorList=[RpcError [message=Optimistic lock failed., severity=ERROR, errorType=APPLICATION, tag=resource-denied, applicationTag=null, info=null, cause=org.opendaylight.yangtools.yang.data.api.schema.tree.ConflictingModificationAppliedException: Node children was modified by other transaction]]} at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299) at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:272) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:96) at com.google.common.util.concurrent.ForwardingFuture.get(ForwardingFuture.java:69) at org.opendaylight.openflowplugin.common.txchain.TransactionChainManager.submitTransaction(TransactionChainManager.java:180) at org.opendaylight.openflowplugin.common.txchain.TransactionChainManager.initialSubmitWriteTransaction(TransactionChainManager.java:85) at org.opendaylight.openflowplugin.impl.device.DeviceContextImpl.initialSubmitTransaction(DeviceContextImpl.java:241) at org.opendaylight.openflowplugin.impl.statistics.StatisticsContextImpl$4.onSuccess(StatisticsContextImpl.java:363) at org.opendaylight.openflowplugin.impl.statistics.StatisticsContextImpl$4.onSuccess(StatisticsContextImpl.java:355) at com.google.common.util.concurrent.Futures$6.run(Futures.java:1319) at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457) at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156) at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:185) at com.google.common.util.concurrent.Futures$ChainingListenableFuture$1.run(Futures.java:918) at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457) at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156) at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:185) at com.google.common.util.concurrent.Futures$ChainingListenableFuture$1.run(Futures.java:918) at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457) at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156) at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:185) at com.google.common.util.concurrent.Futures$ChainingListenableFuture$1.run(Futures.java:918) at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457) at com.google.common.util.concurrent.Futures$ImmediateFuture.addListener(Futures.java:106) at com.google.common.util.concurrent.Futures$ChainingListenableFuture.run(Futures.java:914) at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457) at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156) at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:185) at com.google.common.util.concurrent.SettableFuture.set(SettableFuture.java:53) at org.opendaylight.openflowplugin.impl.rpc.AbstractRequestContext.setResult(AbstractRequestContext.java:32) at org.opendaylight.openflowplugin.impl.device.listener.MultiMsgCollectorImpl.endCollecting(MultiMsgCollectorImpl.java:70) at org.opendaylight.openflowplugin.impl.device.listener.MultiMsgCollectorImpl.addMultipartMsg(MultiMsgCollectorImpl.java:58) at org.opendaylight.openflowplugin.impl.services.AbstractMultipartRequestCallback.onSuccess(AbstractMultipartRequestCallback.java:54) at org.opendaylight.openflowplugin.impl.services.AbstractMultipartRequestCallback.onSuccess(AbstractMultipartRequestCallback.java:22) at org.opendaylight.openflowjava.protocol.impl.core.connection.OutboundQueueEntry.complete(OutboundQueueEntry.java:121) at org.opendaylight.openflowjava.protocol.impl.core.connection.StackedSegment.completeEntry(StackedSegment.java:113) at org.opendaylight.openflowjava.protocol.impl.core.connection.StackedSegment.pairRequest(StackedSegment.java:153) at org.opendaylight.openflowjava.protocol.impl.core.connection.AbstractStackedOutboundQueue.pairRequest(AbstractStackedOutboundQueue.java:203) at org.opendaylight.openflowjava.protocol.impl.core.connection.AbstractOutboundQueueManager.onMessage(AbstractOutboundQueueManager.java:208) at org.opendaylight.openflowjava.protocol.impl.core.connection.ConnectionAdapterImpl.consumeDeviceMessage(ConnectionAdapterImpl.java:143) at org.opendaylight.openflowjava.protocol.impl.core.connection.AbstractConnectionAdapterStatistics.consume(AbstractConnectionAdapterStatistics.java:66) at org.opendaylight.openflowjava.protocol.impl.core.connection.ConnectionAdapterImpl.consume(ConnectionAdapterImpl.java:44) at org.opendaylight.openflowjava.protocol.impl.core.DelegatingInboundHandler.channelRead(DelegatingInboundHandler.java:43) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:341) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) at org.opendaylight.openflowjava.protocol.impl.core.connection.AbstractOutboundQueueManager.channelRead(AbstractOutboundQueueManager.java:195) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:341) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:341) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:341) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:341) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287) at org.opendaylight.openflowjava.protocol.impl.core.IdleHandler.channelRead(IdleHandler.java:39) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:341) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:341) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:129) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:642) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:565) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:479) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:433) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:748) Caused by: OptimisticLockFailedException{message=Optimistic lock failed., errorList=[RpcError [message=Optimistic lock failed., severity=ERROR, errorType=APPLICATION, tag=resource-denied, applicationTag=null, info=null, cause=org.opendaylight.yangtools.yang.data.api.schema.tree.ConflictingModificationAppliedException: Node children was modified by other transaction]]} at org.opendaylight.controller.cluster.datastore.ShardDataTree.lambda$processNextPendingTransaction$0(ShardDataTree.java:751) at org.opendaylight.controller.cluster.datastore.ShardDataTree.processNextPending(ShardDataTree.java:789) at org.opendaylight.controller.cluster.datastore.ShardDataTree.processNextPendingTransaction(ShardDataTree.java:736) at org.opendaylight.controller.cluster.datastore.ShardDataTree.startCanCommit(ShardDataTree.java:819) at org.opendaylight.controller.cluster.datastore.SimpleShardDataTreeCohort.canCommit(SimpleShardDataTreeCohort.java:90) at org.opendaylight.controller.cluster.datastore.ChainedCommitCohort.canCommit(ChainedCommitCohort.java:58) at org.opendaylight.controller.cluster.datastore.CohortEntry.canCommit(CohortEntry.java:97) at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator.handleCanCommit(ShardCommitCoordinator.java:236) at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator.handleForwardedReadyTransaction(ShardCommitCoordinator.java:114) at org.opendaylight.controller.cluster.datastore.Shard.handleForwardedReadyTransaction(Shard.java:699) at org.opendaylight.controller.cluster.datastore.Shard.handleNonRaftCommand(Shard.java:314) at org.opendaylight.controller.cluster.raft.RaftActor.handleCommand(RaftActor.java:270) at org.opendaylight.controller.cluster.common.actor.AbstractUntypedPersistentActor.onReceiveCommand(AbstractUntypedPersistentActor.java:44) at akka.persistence.UntypedPersistentActor.onReceive(PersistentActor.scala:170) at org.opendaylight.controller.cluster.common.actor.MeteringBehavior.apply(MeteringBehavior.java:104) at akka.actor.ActorCell$$anonfun$become$1.applyOrElse(ActorCell.scala:544) at akka.actor.Actor$class.aroundReceive(Actor.scala:502) at akka.persistence.UntypedPersistentActor.akka$persistence$Eventsourced$$super$aroundReceive(PersistentActor.scala:168) at akka.persistence.Eventsourced$$anon$1.stateReceive(Eventsourced.scala:727) at akka.persistence.Eventsourced$class.aroundReceive(Eventsourced.scala:183) at akka.persistence.UntypedPersistentActor.aroundReceive(PersistentActor.scala:168) at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526) at akka.actor.ActorCell.invoke(ActorCell.scala:495) at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257) at akka.dispatch.Mailbox.run(Mailbox.scala:224) at akka.dispatch.Mailbox.exec(Mailbox.scala:234) at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) Caused by: org.opendaylight.yangtools.yang.data.api.schema.tree.ConflictingModificationAppliedException: Node children was modified by other transaction at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkConflicting(SchemaAwareApplyOperation.java:78) at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkNotConflicting(SchemaAwareApplyOperation.java:108) at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkWriteApplicable(SchemaAwareApplyOperation.java:170) at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:128) at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkChildPreconditions(AbstractNodeContainerModificationStrategy.java:305) at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkMergeApplicable(AbstractNodeContainerModificationStrategy.java:313) at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:131) at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkChildPreconditions(AbstractNodeContainerModificationStrategy.java:305) at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkTouchApplicable(AbstractNodeContainerModificationStrategy.java:288) at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:125) at org.opendaylight.yangtools.yang.data.impl.schema.tree.StructuralContainerModificationStrategy.checkApplicable(StructuralContainerModificationStrategy.java:99) at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkChildPreconditions(AbstractNodeContainerModificationStrategy.java:305) at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkTouchApplicable(AbstractNodeContainerModificationStrategy.java:288) at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:125) at org.opendaylight.yangtools.yang.data.impl.schema.tree.RootModificationApplyOperation.checkApplicable(RootModificationApplyOperation.java:72) at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractDataTreeTip.validate(AbstractDataTreeTip.java:35) at org.opendaylight.controller.cluster.datastore.ShardDataTree.lambda$processNextPendingTransaction$0(ShardDataTree.java:743) 2017-11-09 13:28:33,003 | WARN | ntLoopGroup-7-36 | ContextChainHolderImpl | 307 - org.opendaylight.openflowplugin.impl - 0.4.2.Carbon-redhat-2 | Not able to set MASTER role on device openflow:123674267060747, reason: Initial transaction cannot be submitted. 2017-11-09 13:28:33,003 | WARN | ntLoopGroup-7-36 | ContextChainHolderImpl | 307 - org.opendaylight.openflowplugin.impl - 0.4.2.Carbon-redhat-2 | This mastering is mandatory, destroying context chain and closing connection for device openflow:123674267060747. 2017-11-09 13:28:33,003 | INFO | ntLoopGroup-7-36 | ContextChainImpl | 307 - org.opendaylight.openflowplugin.impl - 0.4.2.Carbon-redhat-2 | Closed clustering services registration for node openflow:123674267060747 </pre> </div></div> <p>3. <em>"Failed to shut down transaction chain for device" (more in 4.):</em></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>2017-11-09 13:28:33,004 | WARN | ntLoopGroup-7-36 | DeviceContextImpl | 307 - org.opendaylight.openflowplugin.impl - 0.4.2.Carbon-redhat-2 | Failed to shut down transaction chain for device openflow:123674267060747: java.util.concurrent.ExecutionException: OptimisticLockFailedException{message=Optimistic lock failed., errorList=[RpcError [message=Optimistic lock failed., severity=ERROR, errorType=APPLICATION, tag=resource-denied, applicationTag=null, info=null, cause=org.opendaylight.yangtools.yang.data.api.schema.tree.ConflictingModificationAppliedException: Node children was modified by other transaction]]}</pre> </div></div> <p>4. <em>"ShardLeaderNotRespondingException: Could not create a READ_WRITE transaction on shard default. The shard leader isn't responding." &amp; "Failed to shut down transaction chain for device openflow:":</em></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>2017-11-09 20:14:30,886 | ERROR | lt-dispatcher-46 | TransactionChainProxy | 211 - org.opendaylight.controller.sal-distributed-datastore - 1.5.2.Carbon-redhat-2 | Tx: member-1-datastore-operational-fe-1-chn-122-txn-86-0 - ready future failed f or previous Tx member-1-datastore-operational-fe-1-chn-122-txn-86-0 2017-11-09 20:14:30,886 | WARN | lt-dispatcher-78 | ConcurrentDOMDataBroker | 211 - org.opendaylight.controller.sal-distributed-datastore - 1.5.2.Carbon-redhat-2 | Tx: DOM-CHAIN-121-85 Error during phase CAN_COMMIT, starting Abort akka.pattern.AskTimeoutException: Ask timed out on [ActorSelection[Anchor(akka.tcp://opendaylight-cluster-data@172.16.0.22:2550/), Path(/user/shardmanager-operational/member-0-shard-default-operational/shard-default-member-1:datastore-operational@1:122-8 5_4948293#1836430803)]] after [30000 ms]. Sender[null] sent message of type "org.opendaylight.controller.cluster.datastore.messages.BatchedModifications". at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:604)[185:com.typesafe.akka.actor:2.4.18] at akka.actor.Scheduler$$anon$4.run(Scheduler.scala:126)[185:com.typesafe.akka.actor:2.4.18] at scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:601)[181:org.scala-lang.scala-library:2.11.11.v20170413-090219-8a413ba7cc] at scala.concurrent.BatchingExecutor$class.execute(BatchingExecutor.scala:109)[181:org.scala-lang.scala-library:2.11.11.v20170413-090219-8a413ba7cc] at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:599)[181:org.scala-lang.scala-library:2.11.11.v20170413-090219-8a413ba7cc] at akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:329)[185:com.typesafe.akka.actor:2.4.18] at akka.actor.LightArrayRevolverScheduler$$anon$4.executeBucket$1(LightArrayRevolverScheduler.scala:280)[185:com.typesafe.akka.actor:2.4.18] at akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:284)[185:com.typesafe.akka.actor:2.4.18] at akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:236)[185:com.typesafe.akka.actor:2.4.18] at java.lang.Thread.run(Thread.java:748)[:1.8.0_151] 2017-11-09 20:14:30,886 | WARN | CommitFutures-64 | TransactionChainManager | 304 - org.opendaylight.openflowplugin.common - 0.4.2.Carbon-redhat-2 | Transaction chain failed, recreating chain due to TransactionCommitFailedException{message=canCommit encountered an unexpected failure, errorList=[RpcError [message=canCommit encountered an unexpected failure, severity=ERROR, errorType=APPLICATION, tag=operation-failed, applicationTag=null, info=null, cause=akka.pattern.AskTimeoutException: Ask timed out on [ActorSelection[Anchor(akka.tcp://opendaylight-cluster-data@172.16.0.22:2550/), Path(/user/shardmanager-operational/member-0-shard-default-operational/shard-default-member-1:datastore-operational@1:122-85_4948293#1836430803)]] after [30000 ms]. Sender[null] sent message of type "org.opendaylight.controller.cluster.datastore.messages.BatchedModifications".]]} at org.opendaylight.controller.md.sal.dom.broker.impl.TransactionCommitFailedExceptionMapper.newWithCause(TransactionCommitFailedExceptionMapper.java:37)[171:org.opendaylight.controller.sal-broker-impl:1.5.2.Carbon-redhat-2] at org.opendaylight.controller.md.sal.dom.broker.impl.TransactionCommitFailedExceptionMapper.newWithCause(TransactionCommitFailedExceptionMapper.java:18)[171:org.opendaylight.controller.sal-broker-impl:1.5.2.Carbon-redhat-2] at org.opendaylight.yangtools.util.concurrent.ExceptionMapper.apply(ExceptionMapper.java:102)[69:org.opendaylight.yangtools.util:1.1.2.Carbon-redhat-2] at org.opendaylight.controller.cluster.databroker.ConcurrentDOMDataBroker.handleException(ConcurrentDOMDataBroker.java:231)[211:org.opendaylight.controller.sal-distributed-datastore:1.5.2.Carbon-redhat-2] at org.opendaylight.controller.cluster.databroker.ConcurrentDOMDataBroker.access$000(ConcurrentDOMDataBroker.java:46)[211:org.opendaylight.controller.sal-distributed-datastore:1.5.2.Carbon-redhat-2] at org.opendaylight.controller.cluster.databroker.ConcurrentDOMDataBroker$1.onFailure(ConcurrentDOMDataBroker.java:126)[211:org.opendaylight.controller.sal-distributed-datastore:1.5.2.Carbon-redhat-2] at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310)[65:com.google.guava:18.0.0] at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)[65:com.google.guava:18.0.0] at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)[65:com.google.guava:18.0.0] at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)[65:com.google.guava:18.0.0] at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)[65:com.google.guava:18.0.0] at com.google.common.util.concurrent.SettableFuture.setException(SettableFuture.java:68)[65:com.google.guava:18.0.0] at org.opendaylight.controller.cluster.datastore.SingleCommitCohortProxy$1.onComplete(SingleCommitCohortProxy.java:58)[211:org.opendaylight.controller.sal-distributed-datastore:1.5.2.Carbon-redhat-2] at akka.dispatch.OnComplete.internal(Future.scala:258)[185:com.typesafe.akka.actor:2.4.18] at akka.dispatch.OnComplete.internal(Future.scala:256)[185:com.typesafe.akka.actor:2.4.18] at akka.dispatch.japi$CallbackBridge.apply(Future.scala:186)[185:com.typesafe.akka.actor:2.4.18] at akka.dispatch.japi$CallbackBridge.apply(Future.scala:183)[185:com.typesafe.akka.actor:2.4.18] at akka.dispatch.japi$CallbackBridge.apply(Future.scala:183)[185:com.typesafe.akka.actor:2.4.18] at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:36)[181:org.scala-lang.scala-library:2.11.11.v20170413-090219-8a413ba7cc] at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)[185:com.typesafe.akka.actor:2.4.18] at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:91)[185:com.typesafe.akka.actor:2.4.18] at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)[185:com.typesafe.akka.actor:2.4.18] at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)[185:com.typesafe.akka.actor:2.4.18] at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)[181:org.scala-lang.scala-library:2.11.11.v20170413-090219-8a413ba7cc] at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:90)[185:com.typesafe.akka.actor:2.4.18] at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:39)[185:com.typesafe.akka.actor:2.4.18] at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:415)[185:com.typesafe.akka.actor:2.4.18] at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)[181:org.scala-lang.scala-library:2.11.11.v20170413-090219-8a413ba7cc] at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)[181:org.scala-lang.scala-library:2.11.11.v20170413-090219-8a413ba7cc] at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)[181:org.scala-lang.scala-library:2.11.11.v20170413-090219-8a413ba7cc] at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)[181:org.scala-lang.scala-library:2.11.11.v20170413-090219-8a413ba7cc] Caused by: akka.pattern.AskTimeoutException: Ask timed out on [ActorSelection[Anchor(akka.tcp://opendaylight-cluster-data@172.16.0.22:2550/), Path(/user/shardmanager-operational/member-0-shard-default-operational/shard-default-member-1:datastore-operational@1:122-85_4948293#1836430803)]] after [30000 ms]. Sender[null] sent message of type "org.opendaylight.controller.cluster.datastore.messages.BatchedModifications". at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:604)[185:com.typesafe.akka.actor:2.4.18] at akka.actor.Scheduler$$anon$4.run(Scheduler.scala:126)[185:com.typesafe.akka.actor:2.4.18] at scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:601)[181:org.scala-lang.scala-library:2.11.11.v20170413-090219-8a413ba7cc] at scala.concurrent.BatchingExecutor$class.execute(BatchingExecutor.scala:109)[181:org.scala-lang.scala-library:2.11.11.v20170413-090219-8a413ba7cc] at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:599)[181:org.scala-lang.scala-library:2.11.11.v20170413-090219-8a413ba7cc] at akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:329)[185:com.typesafe.akka.actor:2.4.18] at akka.actor.LightArrayRevolverScheduler$$anon$4.executeBucket$1(LightArrayRevolverScheduler.scala:280)[185:com.typesafe.akka.actor:2.4.18] at akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:284)[185:com.typesafe.akka.actor:2.4.18] at akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:236)[185:com.typesafe.akka.actor:2.4.18] at java.lang.Thread.run(Thread.java:748)[:1.8.0_151] 2017-11-09 20:14:30,886 | ERROR | CommitFutures-64 | TransactionChainManager | 304 - org.opendaylight.openflowplugin.common - 0.4.2.Carbon-redhat-2 | Transaction commit failed. TransactionCommitFailedException{message=canCommit encountered an unexpected failure, errorList=[RpcError [message=canCommit encountered an unexpected failure, severity=ERROR, errorType=APPLICATION, tag=operation-failed, applicationTag=null, info=null, cause=akka.pattern.AskTimeoutException: Ask timed out on [ActorSelection[Anchor(akka.tcp://opendaylight-cluster-data@172.16.0.22:2550/), Path(/user/shardmanager-operational/member-0-shard-default-operational/shard-default-member-1:datastore-operational@1:1 122-85_4948293#1836430803)]] after [30000 ms]. Sender[null] sent message of type "org.opendaylight.controller.cluster.datastore.messages.BatchedModifications".]]} at org.opendaylight.controller.md.sal.dom.broker.impl.TransactionCommitFailedExceptionMapper.newWithCause(TransactionCommitFailedExceptionMapper.java:37)[171:org.opendaylight.controller.sal-broker-impl:1.5.2.Carbon-redhat-2] at org.opendaylight.controller.md.sal.dom.broker.impl.TransactionCommitFailedExceptionMapper.newWithCause(TransactionCommitFailedExceptionMapper.java:18)[171:org.opendaylight.controller.sal-broker-impl:1.5.2.Carbon-redhat-2] at org.opendaylight.yangtools.util.concurrent.ExceptionMapper.apply(ExceptionMapper.java:102)[69:org.opendaylight.yangtools.util:1.1.2.Carbon-redhat-2] at org.opendaylight.controller.cluster.databroker.ConcurrentDOMDataBroker.handleException(ConcurrentDOMDataBroker.java:231)[211:org.opendaylight.controller.sal-distributed-datastore:1.5.2.Carbon-redhat-2] at org.opendaylight.controller.cluster.databroker.ConcurrentDOMDataBroker.access$000(ConcurrentDOMDataBroker.java:46)[211:org.opendaylight.controller.sal-distributed-datastore:1.5.2.Carbon-redhat-2] at org.opendaylight.controller.cluster.databroker.ConcurrentDOMDataBroker$1.onFailure(ConcurrentDOMDataBroker.java:126)[211:org.opendaylight.controller.sal-distributed-datastore:1.5.2.Carbon-redhat-2] at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310)[65:com.google.guava:18.0.0] at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)[65:com.google.guava:18.0.0] at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)[65:com.google.guava:18.0.0] at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)[65:com.google.guava:18.0.0] at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)[65:com.google.guava:18.0.0] at com.google.common.util.concurrent.SettableFuture.setException(SettableFuture.java:68)[65:com.google.guava:18.0.0] at org.opendaylight.controller.cluster.datastore.SingleCommitCohortProxy$1.onComplete(SingleCommitCohortProxy.java:58)[211:org.opendaylight.controller.sal-distributed-datastore:1.5.2.Carbon-redhat-2] at akka.dispatch.OnComplete.internal(Future.scala:258)[185:com.typesafe.akka.actor:2.4.18] at akka.dispatch.OnComplete.internal(Future.scala:256)[185:com.typesafe.akka.actor:2.4.18] at akka.dispatch.japi$CallbackBridge.apply(Future.scala:186)[185:com.typesafe.akka.actor:2.4.18] at akka.dispatch.japi$CallbackBridge.apply(Future.scala:183)[185:com.typesafe.akka.actor:2.4.18] at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:36)[181:org.scala-lang.scala-library:2.11.11.v20170413-090219-8a413ba7cc] at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)[185:com.typesafe.akka.actor:2.4.18] at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:91)[185:com.typesafe.akka.actor:2.4.18] at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)[185:com.typesafe.akka.actor:2.4.18] at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)[185:com.typesafe.akka.actor:2.4.18] at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)[181:org.scala-lang.scala-library:2.11.11.v20170413-090219-8a413ba7cc] at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:90)[185:com.typesafe.akka.actor:2.4.18] at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:39)[185:com.typesafe.akka.actor:2.4.18] at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:415)[185:com.typesafe.akka.actor:2.4.18] at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)[181:org.scala-lang.scala-library:2.11.11.v20170413-090219-8a413ba7cc] at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)[181:org.scala-lang.scala-library:2.11.11.v20170413-090219-8a413ba7cc] at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)[181:org.scala-lang.scala-library:2.11.11.v20170413-090219-8a413ba7cc] at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)[181:org.scala-lang.scala-library:2.11.11.v20170413-090219-8a413ba7cc] Caused by: akka.pattern.AskTimeoutException: Ask timed out on [ActorSelection[Anchor(akka.tcp://opendaylight-cluster-data@172.16.0.22:2550/), Path(/user/shardmanager-operational/member-0-shard-default-operational/shard-default-member-1:datastore-operational@1:122-85_4948293#1836430803)]] after [30000 ms]. Sender[null] sent message of type "org.opendaylight.controller.cluster.datastore.messages.BatchedModifications". : at scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:601)[181:org.scala-lang.scala-library:2.11.11.v20170413-090219-8a413ba7cc] at scala.concurrent.BatchingExecutor$class.execute(BatchingExecutor.scala:109)[181:org.scala-lang.scala-library:2.11.11.v20170413-090219-8a413ba7cc] at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:599)[181:org.scala-lang.scala-library:2.11.11.v20170413-090219-8a413ba7cc] at akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:329)[185:com.typesafe.akka.actor:2.4.18] at akka.actor.LightArrayRevolverScheduler$$anon$4.executeBucket$1(LightArrayRevolverScheduler.scala:280)[185:com.typesafe.akka.actor:2.4.18] at akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:284)[185:com.typesafe.akka.actor:2.4.18] at akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:236)[185:com.typesafe.akka.actor:2.4.18] at java.lang.Thread.run(Thread.java:748)[:1.8.0_151] 2017-11-09 20:14:30,886 | ERROR | CommitFutures-64 | TransactionChainManager | 304 - org.opendaylight.openflowplugin.common - 0.4.2.Carbon-redhat-2 | Transaction commit failed. TransactionCommitFailedException{message=canCommit encountered an unexpected failure, errorList=[RpcError [message=canCommit encountered an unexpected failure, severity=ERROR, errorType=APPLICATION, tag=operation-failed, applicationTag=null, info=null, cause=akka.pattern.AskTimeoutException: Ask timed out on [ActorSelection[Anchor(akka.tcp://opendaylight-cluster-data@172.16.0.22:2550/), Path(/user/shardmanager-operational/member-0-shard-default-operational/shard-default-member-1:datastore-operational@1:122-85_4948293#1836430803)]] after [30000 ms]. Sender[null] sent message of type "org.opendaylight.controller.cluster.datastore.messages.BatchedModifications".]]} at org.opendaylight.controller.md.sal.dom.broker.impl.TransactionCommitFailedExceptionMapper.newWithCause(TransactionCommitFailedExceptionMapper.java:37)[171:org.opendaylight.controller.sal-broker-impl:1.5.2.Carbon-redhat-2] at org.opendaylight.controller.md.sal.dom.broker.impl.TransactionCommitFailedExceptionMapper.newWithCause(TransactionCommitFailedExceptionMapper.java:18)[171:org.opendaylight.controller.sal-broker-impl:1.5.2.Carbon-redhat-2] at org.opendaylight.yangtools.util.concurrent.ExceptionMapper.apply(ExceptionMapper.java:102)[69:org.opendaylight.yangtools.util:1.1.2.Carbon-redhat-2] at org.opendaylight.controller.cluster.databroker.ConcurrentDOMDataBroker.handleException(ConcurrentDOMDataBroker.java:231)[211:org.opendaylight.controller.sal-distributed-datastore:1.5.2.Carbon-redhat-2] at org.opendaylight.controller.cluster.databroker.ConcurrentDOMDataBroker.access$000(ConcurrentDOMDataBroker.java:46)[211:org.opendaylight.controller.sal-distributed-datastore:1.5.2.Carbon-redhat-2] at org.opendaylight.controller.cluster.databroker.ConcurrentDOMDataBroker$1.onFailure(ConcurrentDOMDataBroker.java:126)[211:org.opendaylight.controller.sal-distributed-datastore:1.5.2.Carbon-redhat-2] at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310)[65:com.google.guava:18.0.0] at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)[65:com.google.guava:18.0.0] at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)[65:com.google.guava:18.0.0] at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)[65:com.google.guava:18.0.0] at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)[65:com.google.guava:18.0.0] at com.google.common.util.concurrent.SettableFuture.setException(SettableFuture.java:68)[65:com.google.guava:18.0.0] at org.opendaylight.controller.cluster.datastore.SingleCommitCohortProxy$1.onComplete(SingleCommitCohortProxy.java:58)[211:org.opendaylight.controller.sal-distributed-datastore:1.5.2.Carbon-redhat-2] at akka.dispatch.OnComplete.internal(Future.scala:258)[185:com.typesafe.akka.actor:2.4.18] at akka.dispatch.OnComplete.internal(Future.scala:256)[185:com.typesafe.akka.actor:2.4.18] at akka.dispatch.japi$CallbackBridge.apply(Future.scala:186)[185:com.typesafe.akka.actor:2.4.18] at akka.dispatch.japi$CallbackBridge.apply(Future.scala:183)[185:com.typesafe.akka.actor:2.4.18] at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:36)[181:org.scala-lang.scala-library:2.11.11.v20170413-090219-8a413ba7cc] at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)[185:com.typesafe.akka.actor:2.4.18] at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:91)[185:com.typesafe.akka.actor:2.4.18] at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)[185:com.typesafe.akka.actor:2.4.18] at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)[185:com.typesafe.akka.actor:2.4.18] (...) 2017-11-09 20:14:32,467 | WARN | lt-dispatcher-19 | ConcurrentDOMDataBroker | 211 - org.opendaylight.controller.sal-distributed-datastore - 1.5.2.Carbon-redhat-2 | Tx: DOM-CHAIN-171-0 Error during phase CAN_COMMIT, starting Abort org.opendaylight.controller.cluster.datastore.exceptions.ShardLeaderNotRespondingException: Could not create a READ_WRITE transaction on shard default. The shard leader isn't responding. at org.opendaylight.controller.cluster.datastore.RemoteTransactionContextSupport.createTransactionContext(RemoteTransactionContextSupport.java:213)[211:org.opendaylight.controller.sal-distributed-datastore:1.5.2.Carbon-redhat-2] at org.opendaylight.controller.cluster.datastore.RemoteTransactionContextSupport.onCreateTransactionComplete(RemoteTransactionContextSupport.java:195)[211:org.opendaylight.controller.sal-distributed-datastore:1.5.2.Carbon-redhat-2] at org.opendaylight.controller.cluster.datastore.RemoteTransactionContextSupport.access$000(RemoteTransactionContextSupport.java:39)[211:org.opendaylight.controller.sal-distributed-datastore:1.5.2.Carbon-redhat-2] at org.opendaylight.controller.cluster.datastore.RemoteTransactionContextSupport$1.onComplete(RemoteTransactionContextSupport.java:135)[211:org.opendaylight.controller.sal-distributed-datastore:1.5.2.Carbon-redhat-2] (...) Caused by: akka.pattern.AskTimeoutException: Ask timed out on [ActorSelection[Anchor(akka.tcp://opendaylight-cluster-data@172.16.0.22:2550/), Path(/user/shardmanager-operational/member-0-shard-default-operational)]] after [5000 ms]. Sender[null] sent message of type "org.opendaylight.controller.cluster.datastore.messages.CreateTransaction". at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:604)[185:com.typesafe.akka.actor:2.4.18] (...) 2017-11-09 20:14:32,467 | WARN | CommitFutures-66 | PingPongTransactionChain | 171 - org.opendaylight.controller.sal-broker-impl - 1.5.2.Carbon-redhat-2 | Transaction chain org.opendaylight.controller.cluster.databroker.DOMBrokerTransactionChain@5770b26 failed with no pending transactions (...) 2017-11-09 20:14:34,170 | WARN | ntLoopGroup-7-40 | DeviceContextImpl | 307 - org.opendaylight.openflowplugin.impl - 0.4.2.Carbon-redhat-2 | Failed to shut down transaction chain for device openflow:123674267060747: java.util.concurrent.TimeoutException: Timeout waiting for task. </pre> </div></div> <p>I'll reach out to openflowplugin-dev &amp; controller-dev again to see if any of these means anything to them...</p> <p>&gt; I'll reach out to openflowplugin-dev &amp; controller-dev again to see if any of these means anything to them...</p> <p>On <a href="https://lists.opendaylight.org/pipermail/controller-dev/2017-November/014064.html" class="external-link" target="_blank" rel="nofollow noopener">https://lists.opendaylight.org/pipermail/controller-dev/2017-November/014064.html</a> <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=rovarga" class="user-hover" rel="rovarga">rovarga</a> says "a large FrontendMetadata part could indicate dangling<br/> transactions or similar".</p> <p>The attached open-transactions_controller-*.txt, produced by the trace:transaction CLI Command (see <a href="https://jira.opendaylight.org/browse/CONTROLLER-1760" title="Tooling to find the real root cause culprit of memory leaks related to non-closed transactions (and tx chains)" class="issue-link" data-issue-key="CONTROLLER-1760"><del>CONTROLLER-1760</del></a>), identify a number of "transaction leaks". I had raised at least some of them already in <a href="https://jira.opendaylight.org/browse/NETVIRT-883" title="Umbrella parent issue for grouping all suspected transaction leaks" class="issue-link" data-issue-key="NETVIRT-883"><del>NETVIRT-883</del></a>, and will take this issue as to opportunity to fix as many of them as I can (with reasonable effort), hoping that ths will help fix this problem... I'm NOT 100% sure it will, but probably worth a try getting them in and then re-testing this (and otherwise it's useful clean up to do, anyway).</p> <p><a href="https://git.opendaylight.org/gerrit/#/q/topic:NETVIRT-985" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/q/topic:NETVIRT-985</a> ...</p> <p>Once everything that is open now, PLUS what <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=skitt" class="user-hover" rel="skitt">skitt</a> will help me with in <a href="https://jira.opendaylight.org/browse/NETVIRT-1000" title="Transaction Leaks in neutronvpn, fibmanager &amp; vpnmanager" class="issue-link" data-issue-key="NETVIRT-1000"><del>NETVIRT-1000</del></a>, are available in stable/carbon, then <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=smalleni" class="user-hover" rel="smalleni">smalleni</a> can retest. (Full disclosure: There are few minor Tx leaks, showing up with much fewer transactions than the biggies, which we'll fix only on master, not on carbon &amp; nitrogen.)</p> <p>Everything done here is now merged to master &amp; carbon; and nitrogen should get merged in the coming days (stable/nitrogen was re-open'd today). In attached Controller<span class="error">&#91;1-3&#93;</span>_open-transactions.txt from latest test with a build including these fixes shows, we can clearly see that all the "big" leaks (as in with hundreds of open TXs) have been plugged; as expected. Preliminary early feedback indicates that they are not hitting an OOM anymore.</p> <p> <br/> There ARE still some open TX, but relatively few, compared to where we started; I don't think it's worth following up on them, in the short term. In case we ever want to look at this again in the future, see new attachments.</p> <p>Testing of another scenario ("nova boot") has hit an OOM that looks exactly this again (another huge 857 MB of 1.6 GB Map inside the MD SAL ShardDataTree), so clogging these TX leaks is likely going to be more of an ongoing repetitive than a one time action..</p> <p>Basically, any time we test new scenarios at scale on a path that hasn't been threaded before, if we hit an OOM that shows a blown up ShardDataTree, we have to <tt>run trace:transactions</tt> and clog more non-closed TXs - best by using the <tt>ManagedTransactionRunner</tt>, continuing on the repetitive pattern of fixes which the patches of the past 2 weeks on the <a href="https://git.opendaylight.org/gerrit/#/q/topic:NETVIRT-985" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/q/topic:NETVIRT-985</a> have shown.</p> <p>Closing this issue to avoid confusing, carrying on more similar work in different places in <a href="https://jira.opendaylight.org/browse/OVSDB-435" title="New OOM due to more TX leaks seen in the nova-boot scenario" class="issue-link" data-issue-key="OVSDB-435"><del>OVSDB-435</del></a>.</p> Blocks NETVIRT-878 GENIUS-102 OPNFLWPLUG-961 OVSDB-435 NETVIRT-883 NETVIRT-1010 NETVIRT-886 OVSDB-425 OVSDB-426 NETVIRT-1000 Duplicate NETVIRT-986 Relates CONTROLLER-1756 CONTROLLER-1787 CONTROLLER-1788 OPNFLWPLUG-955 OPNFLWPLUG-956 OPNFLWPLUG-957 Development Rank 0|i037gf: