Hi All,
We have been doing some scale testing of tunnel creation downstream, and were doing a lot of analysis on where more time is spent.

Basic issue we figured out is, all interface-states were populated fast, however ITM tunnel-state takes a long time to update.
The issue further was narrowed down as interface-state listener events for ITM being triggered very slow. Here are the findings:

1. The various listeners on operational/interface-state across various applications were using the “AsyncDataTreeChangeListenerBase” / “AsyncClusteredDataTreeChangeListenerBase”
In mdsalutil which is having a THREAD_POOL_SIZE=1, and finally all the add(), update(), and remove() methods handling the application logic were on the same thread.
This in turn was delaying all the interface-state listeners, ITM being one among them. We just changed the listener in ITM to a separate thread and avoided all DS reads in the path, and observed 20 mins reduction in the worstcase bringup time.
For further proving the theory, the various listeners in applications were commented out, and what we observed is the worstcase ITM tunnel bringup time changed again reduced drastically.
This is just a root cause analysis, and will ask applications to take calls when and how to fix this. The primary purpose of having AsyncListenerBases in mdsalutil was to take the event from MD-SAL and u take it one at a time, provided you use DatastoreJobCoordinator further to parallelize it.
These are the various listeners in netvirt and genius which has the above mentioned problem:

Genius/itm/itm-impl/src/main/java/org/opendaylight/genius/itm/listeners/InterfaceStateListener.java
Genius/itm/itm-impl/src/main/java/org/opendaylight/genius/itm/monitoring/ItmTunnelEventListener.java
vpnservice/aclservice/impl/src/main/java/org/opendaylight/netvirt/aclservice/listeners/AclInterfaceStateListener.java
vpnservice/elanmanager/elanmanager-impl/src/main/java/org/opendaylight/netvirt/elan/internal/ElanInterfaceStateChangeListener.java
vpnservice/elanmanager/elanmanager-impl/src/main/java/org/opendaylight/netvirt/elan/internal/ElanInterfaceStateClusteredListener.java
vpnservice/ipv6service/impl/src/main/java/org/opendaylight/netvirt/ipv6service/Ipv6ServiceInterfaceEventListener.java
vpnservice/natservice/natservice-impl/src/main/java/org/opendaylight/netvirt/natservice/internal/InterfaceStateEventListener.java
vpnservice/natservice/natservice-impl/src/main/java/org/opendaylight/netvirt/natservice/internal/NatInterfaceStateChangeListener.java
vpnservice/natservice/natservice-impl/src/main/java/org/opendaylight/netvirt/natservice/internal/NatTunnelInterfaceStateListener.java
vpnservice/vpnmanager/vpnmanager-impl/src/main/java/org/opendaylight/netvirt/vpnmanager/InterfaceStateChangeListener.java
vpnservice/vpnmanager/vpnmanager-impl/src/main/java/org/opendaylight/netvirt/vpnmanager/SubnetRouteInterfaceStateChangeListener.java

2. Lot of Datastore reads on the listener processing and no caches.

3. Some of the interface-state listeners are still on DCNs instead of DTCNs.