https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/OPNFLWPLUG-1090 OpenFlowPlugin <p>the cbench csit job is frequently failing because some switches are getting their initial<br/> connection rejected.</p> <p>karaf.log shows this:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> 2020-04-30T22:28:37,238 | WARN | epollEventLoopGroup-9-4 | HandshakeListenerImpl | 310 - org.opendaylight.openflowplugin.impl - 0.10.1 | initial processing failed <span class="code-keyword">for</span> device openflow:3 java.util.ConcurrentModificationException: <span class="code-keyword">null</span> at java.util.HashMap.computeIfAbsent(HashMap.java:1134) ~[?:?] at org.opendaylight.openflowplugin.impl.statistics.ofpspecific.SessionStatistics.getConnectionEvents(SessionStatistics.java:36) ~[310:org.opendaylight.openflowplugin.impl:0.10.1] at org.opendaylight.openflowplugin.impl.statistics.ofpspecific.SessionStatistics.countEvent(SessionStatistics.java:25) ~[310:org.opendaylight.openflowplugin.impl:0.10.1] at org.opendaylight.openflowplugin.impl.connection.listener.HandshakeListenerImpl$1.onSuccess(HandshakeListenerImpl.java:81) [310:org.opendaylight.openflowplugin.impl:0.10.1] at org.opendaylight.openflowplugin.impl.connection.listener.HandshakeListenerImpl$1.onSuccess(HandshakeListenerImpl.java:68) [310:org.opendaylight.openflowplugin.impl:0.10.1] at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1076) [27:com.google.guava:27.1.0.jre] at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) [27:com.google.guava:27.1.0.jre] at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1138) [27:com.google.guava:27.1.0.jre] at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:958) [27:com.google.guava:27.1.0.jre] at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:726) [27:com.google.guava:27.1.0.jre] at com.google.common.util.concurrent.SettableFuture.set(SettableFuture.java:48) [27:com.google.guava:27.1.0.jre] at org.opendaylight.openflowjava.protocol.impl.core.connection.AbstractRpcListener.successfulRpc(AbstractRpcListener.java:91) [317:org.opendaylight.openflowplugin.openflowjava.openflow-protocol-impl:0.10.1] at org.opendaylight.openflowjava.protocol.impl.core.connection.ResponseExpectedRpcListener.completed(ResponseExpectedRpcListener.java:37) [317:org.opendaylight.openflowplugin.openflowjava.openflow-protocol-impl:0.10.1] at org.opendaylight.openflowjava.protocol.impl.core.connection.ConnectionAdapterImpl.consumeDeviceMessage(ConnectionAdapterImpl.java:160) [317:org.opendaylight.openflowplugin.openflowjava.openflow-protocol-impl:0.10.1] at org.opendaylight.openflowjava.protocol.impl.core.connection.AbstractConnectionAdapterStatistics.consume(AbstractConnectionAdapterStatistics.java:68) [317:org.opendaylight.openflowplugin.openflowjava.openflow-protocol-impl:0.10.1] at org.opendaylight.openflowjava.protocol.impl.core.connection.ConnectionAdapterImpl.consume(ConnectionAdapterImpl.java:48) [317:org.opendaylight.openflowplugin.openflowjava.openflow-protocol-impl:0.10.1] at org.opendaylight.openflowjava.protocol.impl.core.DelegatingInboundHandler.channelRead(DelegatingInboundHandler.java:44) [317:org.opendaylight.openflowplugin.openflowjava.openflow-protocol-impl:0.10.1] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [57:io.netty.transport:4.1.48.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [57:io.netty.transport:4.1.48.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [57:io.netty.transport:4.1.48.Final] at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) [55:io.netty.handler:4.1.48.Final] at org.opendaylight.openflowjava.protocol.impl.core.IdleHandler.channelRead(IdleHandler.java:37) [317:org.opendaylight.openflowplugin.openflowjava.openflow-protocol-impl:0.10.1] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [57:io.netty.transport:4.1.48.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [57:io.netty.transport:4.1.48.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [57:io.netty.transport:4.1.48.Final] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) [52:io.netty.codec:4.1.48.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [57:io.netty.transport:4.1.48.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [57:io.netty.transport:4.1.48.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [57:io.netty.transport:4.1.48.Final] at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) [52:io.netty.codec:4.1.48.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) [52:io.netty.codec:4.1.48.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [57:io.netty.transport:4.1.48.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [57:io.netty.transport:4.1.48.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [57:io.netty.transport:4.1.48.Final] at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) [52:io.netty.codec:4.1.48.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) [52:io.netty.codec:4.1.48.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [57:io.netty.transport:4.1.48.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [57:io.netty.transport:4.1.48.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [57:io.netty.transport:4.1.48.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [57:io.netty.transport:4.1.48.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [57:io.netty.transport:4.1.48.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [57:io.netty.transport:4.1.48.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [57:io.netty.transport:4.1.48.Final] at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) [58:io.netty.transport-<span class="code-keyword">native</span>-epoll:4.1.48.Final] at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) [58:io.netty.transport-<span class="code-keyword">native</span>-epoll:4.1.48.Final] at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) [58:io.netty.transport-<span class="code-keyword">native</span>-epoll:4.1.48.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [54:io.netty.common:4.1.48.Final] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [54:io.netty.common:4.1.48.Final] at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [54:io.netty.common:4.1.48.Final] at java.lang.<span class="code-object">Thread</span>.run(<span class="code-object">Thread</span>.java:834) [?:?] 2 </pre> </div></div> <p>that snippet came from <a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/openflowplugin-csit-1node-cbench-only-magnesium/229/odl_1/odl1_karaf.log.gz" class="external-link" target="_blank" rel="nofollow noopener">this job</a></p> <p>Also, in the same log, the below exception is seen. If it's unrelated we may need to open<br/> a new bug. It's definitely coming from a different code path.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> 2020-04-30T22:28:37,377 | WARN | opendaylight-cluster-data-akka.actor.<span class="code-keyword">default</span>-dispatcher-35 | ClusterSingletonServiceGroupImpl | 275 - org.opendaylight.mdsal.singleton-dom-impl - 5.0.10 | Service group openflow:10 service org.opendaylight.openflowplugin.impl.lifecycle.ContextChainImpl@5b477b89 failed to stop, attempting to <span class="code-keyword">continue</span> java.util.ConcurrentModificationException: <span class="code-keyword">null</span> at java.util.AbstractList$RandomAccessSpliterator.get(AbstractList.java:737) ~[?:?] at java.util.AbstractList$RandomAccessSpliterator.forEachRemaining(AbstractList.java:720) ~[?:?] at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) ~[?:?] at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) ~[?:?] at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913) ~[?:?] at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?] at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578) ~[?:?] at org.opendaylight.openflowplugin.impl.lifecycle.ContextChainImpl.closeServiceInstance(ContextChainImpl.java:91) ~[?:?] at org.opendaylight.mdsal.singleton.dom.impl.ClusterSingletonServiceGroupImpl.ensureStopping(ClusterSingletonServiceGroupImpl.java:675) ~[?:?] at org.opendaylight.mdsal.singleton.dom.impl.ClusterSingletonServiceGroupImpl.ensureServicesStopping(ClusterSingletonServiceGroupImpl.java:657) ~[?:?] at org.opendaylight.mdsal.singleton.dom.impl.ClusterSingletonServiceGroupImpl.tryReconcileState(ClusterSingletonServiceGroupImpl.java:571) ~[?:?] at org.opendaylight.mdsal.singleton.dom.impl.ClusterSingletonServiceGroupImpl.reconcileState(ClusterSingletonServiceGroupImpl.java:460) ~[?:?] at org.opendaylight.mdsal.singleton.dom.impl.ClusterSingletonServiceGroupImpl.ownershipChanged(ClusterSingletonServiceGroupImpl.java:341) ~[?:?] at org.opendaylight.mdsal.singleton.dom.impl.AbstractClusterSingletonServiceProviderImpl.ownershipChanged(AbstractClusterSingletonServiceProviderImpl.java:236) ~[?:?] at org.opendaylight.mdsal.singleton.dom.impl.DOMClusterSingletonServiceProviderImpl.ownershipChanged(DOMClusterSingletonServiceProviderImpl.java:22) ~[?:?] at org.opendaylight.controller.cluster.entityownership.EntityOwnershipListenerActor.onEntityOwnershipChanged(EntityOwnershipListenerActor.java:45) ~[223:org.opendaylight.controller.sal-distributed-eos:1.11.1] at org.opendaylight.controller.cluster.entityownership.EntityOwnershipListenerActor.handleReceive(EntityOwnershipListenerActor.java:34) ~[223:org.opendaylight.controller.sal-distributed-eos:1.11.1] at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:24) [33:com.typesafe.akka.actor:2.5.31] at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:20) [33:com.typesafe.akka.actor:2.5.31] at scala.PartialFunction.applyOrElse(PartialFunction.scala:127) [382:org.scala-lang.scala-library:2.12.11.v20200311-100536-VFINAL-cd8410d] at scala.PartialFunction.applyOrElse$(PartialFunction.scala:126) [382:org.scala-lang.scala-library:2.12.11.v20200311-100536-VFINAL-cd8410d] at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:20) [33:com.typesafe.akka.actor:2.5.31] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:175) [382:org.scala-lang.scala-library:2.12.11.v20200311-100536-VFINAL-cd8410d] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:176) [382:org.scala-lang.scala-library:2.12.11.v20200311-100536-VFINAL-cd8410d] at akka.actor.Actor.aroundReceive(Actor.scala:539) [33:com.typesafe.akka.actor:2.5.31] at akka.actor.Actor.aroundReceive$(Actor.scala:537) [33:com.typesafe.akka.actor:2.5.31] at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:227) [33:com.typesafe.akka.actor:2.5.31] at akka.actor.ActorCell.receiveMessage(ActorCell.scala:614) [33:com.typesafe.akka.actor:2.5.31] at akka.actor.ActorCell.invoke(ActorCell.scala:583) [33:com.typesafe.akka.actor:2.5.31] at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:268) [33:com.typesafe.akka.actor:2.5.31] at akka.dispatch.Mailbox.run(Mailbox.scala:229) [33:com.typesafe.akka.actor:2.5.31] at akka.dispatch.Mailbox.exec(Mailbox.scala:241) [33:com.typesafe.akka.actor:2.5.31] at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) [33:com.typesafe.akka.actor:2.5.31] at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) [33:com.typesafe.akka.actor:2.5.31] at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) [33:com.typesafe.akka.actor:2.5.31] at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) [33:com.typesafe.akka.actor:2.5.31] 2 </pre> </div></div> OPNFLWPLUG-1090

ConcurrentModificationException: null when connecting cbench switches

Bug Highest Resolved Done Gobinath Suganthan Jamo Luhrsen CSIT Tue, 5 May 2020 21:13:27 +0000 Fri, 6 Nov 2020 03:08:03 +0000 Thu, 28 May 2020 09:46:22 +0000 Magnesium-SR1 Aluminium statistics-manager 0 2 <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=jluhrsen" class="user-hover" rel="jluhrsen">jluhrsen</a> I had earlier used "test-openflowplugin-cbench" for running cbench tests for debugging a different cbench failure. Can it be used to reproduce this issue too?</p> <p>yes, and looks like you already did it successfully with the cbench test passing. I'll try to run it a few more times, but good<br/> news so far.</p> <p>The bigger question here is if this issue is something we probably want to get in to Magnesium SR1 or if you know it will<br/> not really be a problem for end-users. Is it possible for production environment switches to run in to this?</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=jluhrsen" class="user-hover" rel="jluhrsen">jluhrsen</a> This error had resulted from the race condition caused because of many switch handshakes happening at the same time. How many switches are involved in cbench tests and how fast are they flapped?</p> <p>I think this could occur in production too when there are many switch handshakes (large scale).</p> <p>This is only 16 switches and the problem is on initial connection. Does not seem like an unreasonable state I guess. If your<br/> fix is ready to go, we can try to get it merged asap and see about a respin for the Magnesium SR1 release.</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=jluhrsen" class="user-hover" rel="jluhrsen">jluhrsen</a> I have updated my patch and added reviewers from OFPlugin. We can merge if one of them reviews and gives +2.</p> Development Rank 0|i03sbz: