[OPNFLWPLUG-958] ConcurrentModificationException: at org.opendaylight.openflowplugin.impl.registry.flow.DeviceFlowRegistryImpl.getExistingKey Created: 20/Nov/17  Updated: 06/Mar/18  Resolved: 06/Mar/18

Status: Resolved
Project: OpenFlowPlugin
Component/s: General
Affects Version/s: None
Fix Version/s: Carbon-SR3

Type: Bug Priority: Medium
Reporter: Sam Hague Assignee: Sunil Kumar M S
Resolution: Done Votes: 0
Labels: csit:3node
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Relates
relates to OPNFLWPLUG-960 writePortStatusMessage Resolved
Epic Link: Clustering Stability

 Description   

Below exception comes out during a 3node test. At this point ODL1 has been taken down and brought back to service. During that bringup the below exception is seen.

https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-3node-openstack-ocata-upstream-stateful-carbon/184/odl_1/odl1_karaf.log.gz

2017-11-20 03:17:20,695 | INFO  | ult-dispatcher-6 | StatisticsContextImpl            | 281 - org.opendaylight.openflowplugin.impl - 0.4.3.SNAPSHOT | Starting statistics context cluster services for node openflow:88256775084985
2017-11-20 03:17:20,696 | ERROR | lt-dispatcher-25 | ExecutionList                    | 57 - com.google.guava - 18.0.0 | RuntimeException while executing runnable com.google.common.util.concurrent.Futures$6@3546a15e with executor INSTANCE
java.util.ConcurrentModificationException
	at com.google.common.collect.HashBiMap$Itr.checkForConcurrentModification(HashBiMap.java:339)[57:com.google.guava:18.0.0]
	at com.google.common.collect.HashBiMap$Itr.hasNext(HashBiMap.java:345)[57:com.google.guava:18.0.0]
	at org.opendaylight.openflowplugin.impl.registry.flow.DeviceFlowRegistryImpl.getExistingKey(DeviceFlowRegistryImpl.java:284)[281:org.opendaylight.openflowplugin.impl:0.4.3.SNAPSHOT]
	at org.opendaylight.openflowplugin.impl.registry.flow.DeviceFlowRegistryImpl.lambda$new$0(DeviceFlowRegistryImpl.java:67)[281:org.opendaylight.openflowplugin.impl:0.4.3.SNAPSHOT]
	at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)[:1.8.0_151]
	at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)[:1.8.0_151]
	at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)[:1.8.0_151]
	at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1380)[:1.8.0_151]
	at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:580)[:1.8.0_151]
	at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:270)[:1.8.0_151]
	at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)[:1.8.0_151]
	at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)[:1.8.0_151]
	at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1380)[:1.8.0_151]
	at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:580)[:1.8.0_151]
	at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:270)[:1.8.0_151]
	at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)[:1.8.0_151]
	at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)[:1.8.0_151]
	at java.util.Collections$2.tryAdvance(Collections.java:4717)[:1.8.0_151]
	at java.util.Collections$2.forEachRemaining(Collections.java:4725)[:1.8.0_151]
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)[:1.8.0_151]
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)[:1.8.0_151]
	at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)[:1.8.0_151]
	at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)[:1.8.0_151]
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)[:1.8.0_151]
	at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)[:1.8.0_151]
	at org.opendaylight.openflowplugin.impl.registry.flow.DeviceFlowRegistryImpl$1.onSuccess(DeviceFlowRegistryImpl.java:132)[281:org.opendaylight.openflowplugin.impl:0.4.3.SNAPSHOT]
	at org.opendaylight.openflowplugin.impl.registry.flow.DeviceFlowRegistryImpl$1.onSuccess(DeviceFlowRegistryImpl.java:120)[281:org.opendaylight.openflowplugin.impl:0.4.3.SNAPSHOT]
	at com.google.common.util.concurrent.Futures$6.run(Futures.java:1319)[57:com.google.guava:18.0.0]
	at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)[57:com.google.guava:18.0.0]
	at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)[57:com.google.guava:18.0.0]
	at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)[57:com.google.guava:18.0.0]
	at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:185)[57:com.google.guava:18.0.0]
	at com.google.common.util.concurrent.Futures$ChainingListenableFuture$1.run(Futures.java:918)[57:com.google.guava:18.0.0]
	at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)[57:com.google.guava:18.0.0]
	at com.google.common.util.concurrent.Futures$ImmediateFuture.addListener(Futures.java:106)[57:com.google.guava:18.0.0]
	at com.google.common.util.concurrent.Futures$ChainingListenableFuture.run(Futures.java:914)[57:com.google.guava:18.0.0]
	at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)[57:com.google.guava:18.0.0]
	at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)[57:com.google.guava:18.0.0]
	at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)[57:com.google.guava:18.0.0]
	at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:185)[57:com.google.guava:18.0.0]
	at com.google.common.util.concurrent.SettableFuture.set(SettableFuture.java:53)[57:com.google.guava:18.0.0]
	at org.opendaylight.controller.cluster.datastore.messages.ReadData.processResponse(ReadData.java:40)[203:org.opendaylight.controller.sal-distributed-datastore:1.5.3.SNAPSHOT]
	at org.opendaylight.controller.cluster.datastore.RemoteTransactionContext$1.onComplete(RemoteTransactionContext.java:187)[203:org.opendaylight.controller.sal-distributed-datastore:1.5.3.SNAPSHOT]
	at akka.dispatch.OnComplete.internal(Future.scala:259)[177:com.typesafe.akka.actor:2.4.18]
	at akka.dispatch.OnComplete.internal(Future.scala:256)[177:com.typesafe.akka.actor:2.4.18]
	at akka.dispatch.japi$CallbackBridge.apply(Future.scala:186)[177:com.typesafe.akka.actor:2.4.18]
	at akka.dispatch.japi$CallbackBridge.apply(Future.scala:183)[177:com.typesafe.akka.actor:2.4.18]
	at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:36)[173:org.scala-lang.scala-library:2.11.11.v20170413-090219-8a413ba7cc]
	at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)[177:com.typesafe.akka.actor:2.4.18]
	at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:91)[177:com.typesafe.akka.actor:2.4.18]
	at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)[177:com.typesafe.akka.actor:2.4.18]
	at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)[177:com.typesafe.akka.actor:2.4.18]
	at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)[173:org.scala-lang.scala-library:2.11.11.v20170413-090219-8a413ba7cc]
	at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:90)[177:com.typesafe.akka.actor:2.4.18]
	at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:39)[177:com.typesafe.akka.actor:2.4.18]
	at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:415)[177:com.typesafe.akka.actor:2.4.18]
	at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)[173:org.scala-lang.scala-library:2.11.11.v20170413-090219-8a413ba7cc]
	at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)[173:org.scala-lang.scala-library:2.11.11.v20170413-090219-8a413ba7cc]
	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)[173:org.scala-lang.scala-library:2.11.11.v20170413-090219-8a413ba7cc]


 Comments   
Comment by Anil Vishnoi [ 12/Dec/17 ]

sunilkumarms is this bug also addressed through the group-id exception bug ?

Comment by Sunil Kumar M S [ 12/Dec/17 ]

Avishnoi this issue is not addressed in the current patch. I will have to start work on this after completion of current task. 

Comment by Sunil Kumar M S [ 13/Dec/17 ]

shague as per my observation on the CSIT runs, this issue is not seen on single node test. It may be appearing in the cluster tests, so can you confirm if this is appearing in cluster tests only? 

Comment by Sam Hague [ 14/Dec/17 ]

Yes, this is a 3node / cluster issue. All those issues I label with the csit:3node label as well as the "Clustering Stability" epic.

Comment by Sunil Kumar M S [ 09/Jan/18 ]

https://git.opendaylight.org/gerrit/#/c/66407/

Comment by Sunil Kumar M S [ 13/Feb/18 ]

Hello Anil, 

please take a look https://git.opendaylight.org/gerrit/#/c/66407/ and let me know if anything else needs to worked on this. I have run CSIT couple of time earlier. But now the build isn't passing on carbon and hence tests are unstable.

Generated at Wed Feb 07 20:33:50 UTC 2024 using Jira 8.20.10#820010-sha1:ace47f9899e9ee25d7157d59aa17ab06aee30d3d.