Clustering: ConcurrentModificationException in TransactionProxy

The following exception was seen:

2015-01-27 10:58:23,674 | ERROR | lt-dispatcher-14 | Dispatcher | 168 - com.typesafe.akka.slf4j - 2.3.4 |
java.util.ConcurrentModificationException
at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:859)[:1.7.0_71]
at java.util.ArrayList$Itr.next(ArrayList.java:831)[:1.7.0_71]
at org.opendaylight.controller.cluster.datastore.TransactionProxy$TransactionFutureCallback.onComplete(TransactionProxy.java:682)[182:org.opendaylight.controller.sal-distributed-datastore:1.1.2.PACNET]
at akka.dispatch.OnComplete.internal(Future.scala:247)[167:com.typesafe.akka.actor:2.3.4]
at akka.dispatch.OnComplete.internal(Future.scala:244)[167:com.typesafe.akka.actor:2.3.4]
at akka.dispatch.japi$CallbackBridge.apply(Future.scala:174)[167:com.typesafe.akka.actor:2.3.4]

From the mailing list:

Here is chain of events that lead to ConcurrentModificationException:

1. TransactionOperation created in TransactionProxy.read method is invoked. Method invoke of this TransactionOperation call Futures.addCallback.
2. Few frames down the stack this call result in com.google.common.util.concurrent.AbstractFuture.addListener being executed, which call executionList.add.
3. Looks like when future finish execution before listener gets added to ListenableFuture, further additions of listeners to ListenableFuture will execute listener methods immediately. Looks like this situation is uncommon, but nevertheless happen from time to time. So, executionList.add call FutureCallback.onSuccess method of anonymous class created in TransactionProxy.read in the same thread in which iteration over txOperationsOnComplete happen.
4. FutureCallback.onSuccess from TransactionProxy.read will update proxyFuture in same thread in which iteration over txOperationsOnComplete happen.
5. This will execute FutureCallback.onSuccess which we wrote (we use Futures.transform on future returned by ReadWriteTransaction.read), and in this listener we call ReadWriteTransaction.put.
6. ReadWriteTransaction.put will result in TransactionFutureCallback.addTxOperationOnComplete being called in the same thread in which iteration over txOperationsOnComplete happen.