Details
-
Improvement
-
Status: Resolved
-
Resolution: Done
-
Helium
-
None
-
None
-
Operating System: All
Platform: All
Description
Currently the WriteTx commit process is all single-theaded. At scale this may be a bottleneck. We should try to off-load and multi-thread at least parts of the commit so as to avoid blocking the commit thread and also gain some parallelism.
One part is the notification of DataChangeListeners. Currently the ThreePhaseCommitImpl calls DataChangeListeners on an executor that executes tasks in the same thread as the caller so DataChangeListeners block the commit thread. This was done because DataChangeListeners need to be notified of change events in the order that they occur.
However we can queue events per DataChangeListener and dispatch them serially on separate threads. I have prototyped a QueuedNotificationManager to do this. In addition, the QueuedNotificationManager optimizes its memory footprint by only allocating and maintaining a queue and executor task for a listener when there are pending notifications. Once all notifications have been dispatched, the queue and task are discarded.
Another part of the commit process that can be off-loaded is the notification of the ListenableFuture that's returned from the submit call. Typically, clients will use Futures#addCallBack so as not to block on the ListenableFuture. The default behavior of Futures#addCallBack is to use MoreExecutors#sameThreadExecutor which results in the client callback running on the commit thread. While it is recommended that clients not do any expensive processing and/or block, I think it's fragile to rely on that. One scenario: if a client does a commit in an RPC call and sets the RPC result Future in the commit callback, if the caller of the RPC also uses Futures#addCallBack with MoreExecutors#sameThreadExecutor, then it will also run in the commit thread and so on.
To off-load commit ListenableFuture callbacks, I have prototyped an AsyncNotifyingListeningExecutorService class that uses an AsyncNotifyingListenableFutureTask that takes an Executor and notifies registered ListenableFuture Runnables on the Executor when the task completes and the future result is set. ListenableFuture#addListener also takes an Executor which, by default, is MoreExecutors#sameThreadExecutor. This Executor is still used but if the AsyncNotifyingListenableFutureTask detects that the listener Runnable task will run on the same thread that completed the AsyncNotifyingListenableFutureTask, then it is off-loaded to the other Executor.