[CONTROLLER-1128] Clustering: ConcurrentModificationException in TransactionProxy Created: 28/Jan/15  Updated: 03/Feb/15  Resolved: 03/Feb/15

Status: Resolved
Project: controller
Component/s: mdsal
Affects Version/s: Post-Helium
Fix Version/s: None

Type: Bug
Reporter: Tom Pantelis Assignee: Tom Pantelis
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Operating System: All
Platform: All


External issue ID: 2650
Priority: Normal

 Description   

The following exception was seen:

2015-01-27 10:58:23,674 | ERROR | lt-dispatcher-14 | Dispatcher | 168 - com.typesafe.akka.slf4j - 2.3.4 |
java.util.ConcurrentModificationException
at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:859)[:1.7.0_71]
at java.util.ArrayList$Itr.next(ArrayList.java:831)[:1.7.0_71]
at org.opendaylight.controller.cluster.datastore.TransactionProxy$TransactionFutureCallback.onComplete(TransactionProxy.java:682)[182:org.opendaylight.controller.sal-distributed-datastore:1.1.2.PACNET]
at akka.dispatch.OnComplete.internal(Future.scala:247)[167:com.typesafe.akka.actor:2.3.4]
at akka.dispatch.OnComplete.internal(Future.scala:244)[167:com.typesafe.akka.actor:2.3.4]
at akka.dispatch.japi$CallbackBridge.apply(Future.scala:174)[167:com.typesafe.akka.actor:2.3.4]

From the mailing list:

Here is chain of events that lead to ConcurrentModificationException:

1. TransactionOperation created in TransactionProxy.read method is invoked. Method invoke of this TransactionOperation call Futures.addCallback.
2. Few frames down the stack this call result in com.google.common.util.concurrent.AbstractFuture.addListener being executed, which call executionList.add.
3. Looks like when future finish execution before listener gets added to ListenableFuture, further additions of listeners to ListenableFuture will execute listener methods immediately. Looks like this situation is uncommon, but nevertheless happen from time to time. So, executionList.add call FutureCallback.onSuccess method of anonymous class created in TransactionProxy.read in the same thread in which iteration over txOperationsOnComplete happen.
4. FutureCallback.onSuccess from TransactionProxy.read will update proxyFuture in same thread in which iteration over txOperationsOnComplete happen.
5. This will execute FutureCallback.onSuccess which we wrote (we use Futures.transform on future returned by ReadWriteTransaction.read), and in this listener we call ReadWriteTransaction.put.
6. ReadWriteTransaction.put will result in TransactionFutureCallback.addTxOperationOnComplete being called in the same thread in which iteration over txOperationsOnComplete happen.



 Comments   
Comment by Tom Pantelis [ 29/Jan/15 ]

Submitted https://git.opendaylight.org/gerrit/#/c/14591/

Generated at Wed Feb 07 19:54:45 UTC 2024 using Jira 8.20.10#820010-sha1:ace47f9899e9ee25d7157d59aa17ab06aee30d3d.