Clustering : Use the 95th percentile number to determine the back pressure for creating transactions

Currently we have a back-pressure mechanism in place which throttles creation of transactions based on how much time 98% of the transactions take. On my macbook which is a 2.3 GHz i7 with 8 Cores and 16GB RAM this takes about 140-160ms for a 3-node cluster. Consequently we are able to get only 6-7 txns through when we have mininet connected with 127 switches.

Based on experimentation I see that if I switch to using the 95th percentile number to throttle transaction creation we can do about 18-20 txn/sec because the 95th percentile number is 40-60ms.

Just for the record on my mac-book I see 75% of the transactions taking 7-8ms. If back-pressure is set to lower levels however we start seeing some instability in the cluster.

Here are couple of modifications I had made when I came up with these observations,

• I set throughput for the default akka dispatcher at 1. This is to make the dispatcher fair. An additional experiment may be to use a different dispatcher for Shards and ShardTransactions.
• I disabled the IsolatedLeaderCheck. Enabling IsolatedLeaderCheck does seem to have an impact on work processing. This is possibly due to the fact that it uses a schedule to schedule tasks at a fixed interval. If we want to continue to use IsolatedLeaderCheck we may want to change the scheduling strategy, for example we could use the heartbeat mechanism to check if a leader has become isolated. Or we could use the scheduleOnce strategy for scheduling tasks so that we only schedule a task when we have received a message. I need to create a separate defect for this.