While running the DistributeDataStoreIntergrationTest repeatedly in a script, I found an intermittent failure that occurred a couple times after 20-30 successful runs.

The failure occurred each time in testWriteTransactionWithMultipleShards with an IllegalStateException stating that the transaction has already been sealed. This means that a modification operation (put/merge/delete) was done after the transaction was submitted. However the sequence of calls in the test is valid, i.e. it simply creates a write Tx, makes 2 write calls and calls ready.

After debugging, I found a timing bug in the TransactionProxy.TransactionFutureCallback inner class. On each operation, it first checks if a Tx has been created in the related shard yet (via a non-null TransactionContext instance), otherwise an TransactionFutureCallback instance is created, which initiates the shard Tx creation async, and the operation is cached in the TransactionFutureCallback to be executed when Tx create finishes. The operations are cached in a List to preserve order with synchronization. However there is a small timing window where the operation method may see a non-null TransactionContext but the TransactionFutureCallback hasn't finished executing the previous operations. In this case, the operation method will execute the operation directly in the TransactionContext. This can cause an operation to be performed out-of-order. In the test failure, the ready operation was performed before the write.

The solution is to only publish the TransactionContext instance in TransactionFutureCallback after it has finished executing the cached operations.