possible deadlocks causing slow feature installations

There have been several complaints on the email lists about a deadlockmonitor warning in the karaf.log. It seems when this happens during feature install, it will take an additional 90 seconds for the deadlockmonitor to restart the service after which point the system does seem to operate properly.

this provides a lesser user experience when it happens.

There may be many/several different flavors of this, but one easy way to
recreate it with a deadlock warning in "distribute-config-datastore-provider":

• unzip the Lithium RCO bits
• start karaf with the clean option
• install the Helium based openflowplugin (feature:install odl-openflowplugin-flow-services-ui")
• <optional> you can connect mininet to verify the plugin is ok
• logout of karaf
• start karaf with the clean option
• install the Lithium based openflowplugin (feature:install odl-openflowplugin-flow-services-ui-li")
• monitor the data/log/karaf.log file for the DeadlockMonitor
• you should see something similar to the below.

2015-06-06 09:39:12,989 | WARN | saction-52-54'}} | DeadlockMonitor | 113 - org.openday
light.controller.config-manager - 0.3.0.Lithium-RC0 | ModuleIdentifier

{factoryName='distributed-config-da tastore-provider', instanceName='distributed-config-store-module'}

did not finish after 89986 ms
2015-06-06 09:39:13,781 | ERROR | config-pusher | DistributedDataStore | 179 - org.openday
light.controller.sal-distributed-datastore - 1.2.0.Lithium-RC0 | Shared leaders failed to settle in 90 se
conds, giving up

As a side note, I have seen this in "binding-broker-impl", although I don't have exact steps to
reproduce that one.

2015-06-02 09:16:46,834 | WARN | saction-50-52'}} | DeadlockMonitor | 113 - org.opendaylight.controller.
config-manager - 0.3.0.SNAPSHOT | ModuleIdentifier

{factoryName='binding-broker-impl', instanceName='binding-broker-impl'}

did not finish after 89990 ms

It's also been reported in "statistics-manager" on the controller-dev email list:

2015-05-26 11:01:34,017 | WARN | saction-33-35'}} | kMonitor$DeadlockMonitorRunnable | 151 - org.opendaylight.controller.config-manager - 0.2.8.Helium-SR3 | ModuleIdentifier

{factoryName='statistics-manager', instanceName='statistics-manager'}

did not finish after 10984 ms