[MDSAL-689] Multiple issues during global context update

Wed, 29 Sep 2021 11:45:08 +0000

Performing a late install operation like this:

opendaylight-user@root>feature:install odl-restconf-nb-rfc8040
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.esotericsoftware.kryo.util.UnsafeUtil (file:/home/nite/odl/netconf/karaf/netconf-karaf-2.0.6-SNAPSHOT/data/cache/org.eclipse.osgi/207/0/bundleFile) to constructor java.nio.DirectByteBuffer(long,int,java.lang.Object)
WARNING: Please consider reporting this to the maintainers of com.esotericsoftware.kryo.util.UnsafeUtil
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
opendaylight-user@root>feature:install odl-mdsal-apidocs

leads to a reactor rewiring via rabbitmq/dropwizard metricts. This is not a problem as such, but we also end up refreshing the global runtime context:

2021-09-20T16:13:12,340 | INFO  | features-3-thread-1 | OSGiModuleInfoSnapshotImpl       | 242 - org.opendaylight.mdsal.dom-schema-osgi - 8.0.5 | EffectiveModelContext generation 2 activated
2021-09-20T16:13:12,498 | INFO  | features-3-thread-1 | OSGiBindingRuntimeContextImpl    | 226 - org.opendaylight.mdsal.binding-runtime-osgi - 8.0.5 | BindingRuntimeContext generation 2 activated
2021-09-20T16:13:12,501 | INFO  | features-3-thread-1 | OSGiBindingDOMCodecServicesImpl  | 220 - org.opendaylight.mdsal.binding-dom-codec-osgi - 8.0.5 | Binding/DOM Codec generation 2 activated

but we then proceed to restart components:

2021-09-20T16:13:12,507 | INFO  | features-3-thread-1 | AbstractAdaptedService           | 217 - org.opendaylight.mdsal.binding-dom-adapter - 8.0.5 | Binding/DOM adapter for ActionProviderService deactivated (reason 5)
2021-09-20T16:13:12,508 | INFO  | features-3-thread-1 | OSGiRemoteOpsProvider            | 213 - org.opendaylight.controller.sal-remoterpc-connector - 4.0.3 | Remote Operations service stopping
2021-09-20T16:13:12,509 | INFO  | features-3-thread-1 | OSGiRemoteOpsProvider            | 213 - org.opendaylight.controller.sal-remoterpc-connector - 4.0.3 | Remote Operations services stopped
2021-09-20T16:13:12,514 | INFO  | features-3-thread-1 | AbstractAdaptedService           | 217 - org.opendaylight.mdsal.binding-dom-adapter - 8.0.5 | Binding/DOM adapter for ActionService deactivated (reason 5)
2021-09-20T16:13:12,531 | INFO  | features-3-thread-1 | OSGiEntityOwnershipService       | 244 - org.opendaylight.mdsal.eos-binding-adapter - 8.0.5 | Entity Ownership Service adapter stopping
2021-09-20T16:13:12,541 | INFO  | features-3-thread-1 | OSGiEntityOwnershipService       | 244 - org.opendaylight.mdsal.eos-binding-adapter - 8.0.5 | Entity Ownership Service adapter stopped
2021-09-20T16:13:12,543 | INFO  | features-3-thread-1 | OSGiClusterAdmin                 | 209 - org.opendaylight.controller.sal-cluster-admin-impl - 4.0.3 | Cluster Admin services stopped
2021-09-20T16:13:12,545 | INFO  | features-3-thread-1 | OSGiClusterSingletonServiceProvider | 253 - org.opendaylight.mdsal.singleton-dom-impl - 8.0.5 | Cluster Singleton Service stopping
2021-09-20T16:13:12,550 | INFO  | features-3-thread-1 | OSGiClusterSingletonServiceProvider | 253 - org.opendaylight.mdsal.singleton-dom-impl - 8.0.5 | Cluster Singleton Service stopped
2021-09-20T16:13:12,558 | INFO  | features-3-thread-1 | AbstractAdaptedService           | 217 - org.opendaylight.mdsal.binding-dom-adapter - 8.0.5 | Binding/DOM adapter for RpcProviderService deactivated (reason 5)
2021-09-20T16:13:12,568 | INFO  | features-3-thread-1 | AbstractAdaptedService           | 217 - org.opendaylight.mdsal.binding-dom-adapter - 8.0.5 | Binding/DOM adapter for RpcConsumerRegistry deactivated (reason 5)
2021-09-20T16:13:12,570 | INFO  | opendaylight-cluster-data-akka.actor.default-dispatcher-15 | LocalActorRef                    | 205 - org.opendaylight.controller.repackaged-akka - 4.0.3 | Message [akka.dispatch.sysmsg.Suspend] from Actor[akka://opendaylight-cluster-data/user/EOSBootstrap/ListenerRegistry/TypeList
ener:type=org.opendaylight.mdsal.AsyncServiceCloseEntityType-5c0e0f2c-6c58-4e3f-ac85-a8d57c4a1d66#-807810114] to Actor[akka://opendaylight-cluster-data/user/EOSBootstrap/ListenerRegistry/TypeListener:type=org.opendaylight.mdsal.AsyncServiceCloseEntityType-5c0e0f2c-6c58-4e3f-ac85-a8d57c4a1d66#-807810114] was not del
ivered. [1] dead letters encountered. If this is not an expected behavior then Actor[akka://opendaylight-cluster-data/user/EOSBootstrap/ListenerRegistry/TypeListener:type=org.opendaylight.mdsal.AsyncServiceCloseEntityType-5c0e0f2c-6c58-4e3f-ac85-a8d57c4a1d66#-807810114] may have terminated unexpectedly. This loggin
g can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
2021-09-20T16:13:12,581 | INFO  | features-3-thread-1 | OSGiDOMRpcRouter                 | 241 - org.opendaylight.mdsal.dom-broker - 8.0.5 | DOM RPC/Action router stopped
2021-09-20T16:13:12,583 | INFO  | features-3-thread-1 | OSGiDistributedDataStore         | 212 - org.opendaylight.controller.sal-distributed-datastore - 4.0.3 | Distributed Datastore type OPERATIONAL stopping
2021-09-20T16:13:12,594 | INFO  | features-3-thread-1 | OSGiH2Store                      | 185 - org.opendaylight.aaa.idm-store-h2 - 0.14.3 | H2 IDMStore deactivated
2021-09-20T16:13:12,606 | INFO  | features-3-thread-1 | OSGiPasswordServiceConfigBootstrap | 187 - org.opendaylight.aaa.password-service-impl - 0.14.3 | No longer listening for password service configuration
2021-09-20T16:13:12,611 | INFO  | features-3-thread-1 | AbstractAdaptedService           | 217 - org.opendaylight.mdsal.binding-dom-adapter - 8.0.5 | Binding/DOM adapter for DataBroker deactivated (reason 5)
2021-09-20T16:13:12,611 | INFO  | features-3-thread-1 | OSGiDOMDataBroker                | 212 - org.opendaylight.controller.sal-distributed-datastore - 4.0.3 | DOM Data Broker stopping
2021-09-20T16:13:12,612 | INFO  | features-3-thread-1 | OSGiDOMDataBroker                | 212 - org.opendaylight.controller.sal-distributed-datastore - 4.0.3 | DOM Data Broker stopped
2021-09-20T16:13:12,619 | INFO  | features-3-thread-1 | OSGiDOMStore                     | 212 - org.opendaylight.controller.sal-distributed-datastore - 4.0.3 | Datastore service type OPERATIONAL deactivated
2021-09-20T16:13:12,620 | INFO  | features-3-thread-1 | AbstractDataStore                | 212 - org.opendaylight.controller.sal-distributed-datastore - 4.0.3 | Closing data store ClientIdentifier{frontend=member-1-frontend-datastore-operational, generation=0}
2021-09-20T16:13:12,626 | INFO  | opendaylight-cluster-data-shard-dispatcher-20 | ShardManager                     | 212 - org.opendaylight.controller.sal-distributed-datastore - 4.0.3 | Shutting down ShardManager shard-manager-operational - waiting on 3 shards
2021-09-20T16:13:12,629 | INFO  | opendaylight-cluster-data-shard-dispatcher-29 | Shard                            | 210 - org.opendaylight.controller.sal-clustering-commons - 4.0.3 | Stopping Shard member-1-shard-inventory-operational
2021-09-20T16:13:12,634 | INFO  | opendaylight-cluster-data-shard-dispatcher-24 | Shard                            | 210 - org.opendaylight.controller.sal-clustering-commons - 4.0.3 | Stopping Shard member-1-shard-topology-operational
2021-09-20T16:13:12,636 | INFO  | opendaylight-cluster-data-shard-dispatcher-23 | Shard                            | 210 - org.opendaylight.controller.sal-clustering-commons - 4.0.3 | Stopping Shard member-1-shard-default-operational
2021-09-20T16:13:12,638 | INFO  | opendaylight-cluster-data-shard-dispatcher-17 | ShardManager                     | 212 - org.opendaylight.controller.sal-distributed-datastore - 4.0.3 | Stopping ShardManager shard-manager-operational
2021-09-20T16:13:12,647 | INFO  | features-3-thread-1 | OSGiDistributedDataStore         | 212 - org.opendaylight.controller.sal-distributed-datastore - 4.0.3 | Distributed Datastore type OPERATIONAL stopped
2021-09-20T16:13:12,647 | INFO  | features-3-thread-1 | OSGiDistributedDataStore         | 212 - org.opendaylight.controller.sal-distributed-datastore - 4.0.3 | Distributed Datastore type CONFIGURATION stopping
2021-09-20T16:13:12,654 | INFO  | opendaylight-cluster-data-akka.actor.default-dispatcher-31 | RepointableActorRef              | 205 - org.opendaylight.controller.repackaged-akka - 4.0.3 | Message [org.opendaylight.controller.cluster.common.actor.ExecuteInSelfMessage] to Actor[akka://opendaylight-cluster-data/user
/shardmanager-operational#-1293111216] was not delivered. [2] dead letters encountered. If this is not an expected behavior then Actor[akka://opendaylight-cluster-data/user/shardmanager-operational#-1293111216] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 
'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
2021-09-20T16:13:12,657 | INFO  | features-3-thread-1 | OSGiDOMStore                     | 212 - org.opendaylight.controller.sal-distributed-datastore - 4.0.3 | Datastore service type CONFIGURATION deactivated
2021-09-20T16:13:12,657 | INFO  | opendaylight-cluster-data-akka.actor.default-dispatcher-30 | RepointableActorRef              | 205 - org.opendaylight.controller.repackaged-akka - 4.0.3 | Message [org.opendaylight.controller.cluster.common.actor.ExecuteInSelfMessage] to Actor[akka://opendaylight-cluster-data/user
/shardmanager-operational#-1293111216] was not delivered. [3] dead letters encountered. If this is not an expected behavior then Actor[akka://opendaylight-cluster-data/user/shardmanager-operational#-1293111216] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 
'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
2021-09-20T16:13:12,659 | INFO  | opendaylight-cluster-data-akka.actor.default-dispatcher-30 | RepointableActorRef              | 205 - org.opendaylight.controller.repackaged-akka - 4.0.3 | Message [org.opendaylight.controller.cluster.databroker.actors.dds.AbstractDataStoreClientBehavior$$Lambda$2370/0x00000007c166
ec40] to Actor[akka://opendaylight-cluster-data/user/$b#1333458643] was not delivered. [4] dead letters encountered. If this is not an expected behavior then Actor[akka://opendaylight-cluster-data/user/$b#1333458643] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration set
tings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
2021-09-20T16:13:12,661 | INFO  | features-3-thread-1 | AbstractDataStore                | 212 - org.opendaylight.controller.sal-distributed-datastore - 4.0.3 | Closing data store ClientIdentifier{frontend=member-1-frontend-datastore-config, generation=0}
2021-09-20T16:13:12,674 | INFO  | opendaylight-cluster-data-shard-dispatcher-17 | ShardManager                     | 212 - org.opendaylight.controller.sal-distributed-datastore - 4.0.3 | Shutting down ShardManager shard-manager-config - waiting on 3 shards
2021-09-20T16:13:12,677 | INFO  | opendaylight-cluster-data-shard-dispatcher-24 | Shard                            | 210 - org.opendaylight.controller.sal-clustering-commons - 4.0.3 | Stopping Shard member-1-shard-topology-config
2021-09-20T16:13:12,678 | INFO  | opendaylight-cluster-data-shard-dispatcher-26 | Shard                            | 210 - org.opendaylight.controller.sal-clustering-commons - 4.0.3 | Stopping Shard member-1-shard-inventory-config
2021-09-20T16:13:12,679 | INFO  | opendaylight-cluster-data-shard-dispatcher-17 | Shard                            | 210 - org.opendaylight.controller.sal-clustering-commons - 4.0.3 | Stopping Shard member-1-shard-default-config
2021-09-20T16:13:12,687 | INFO  | opendaylight-cluster-data-shard-dispatcher-17 | ShardManager                     | 212 - org.opendaylight.controller.sal-distributed-datastore - 4.0.3 | Stopping ShardManager shard-manager-config
2021-09-20T16:13:12,690 | INFO  | features-3-thread-1 | OSGiDistributedDataStore         | 212 - org.opendaylight.controller.sal-distributed-datastore - 4.0.3 | Distributed Datastore type CONFIGURATION stopped
2021-09-20T16:13:12,693 | INFO  | opendaylight-cluster-data-akka.actor.default-dispatcher-4 | RepointableActorRef              | 205 - org.opendaylight.controller.repackaged-akka - 4.0.3 | Message [org.opendaylight.controller.cluster.common.actor.ExecuteInSelfMessage] to Actor[akka://opendaylight-cluster-data/user/
shardmanager-config#795186601] was not delivered. [5] dead letters encountered. If this is not an expected behavior then Actor[akka://opendaylight-cluster-data/user/shardmanager-config#795186601] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-
letters' and 'akka.log-dead-letters-during-shutdown'.
2021-09-20T16:13:12,693 | INFO  | features-3-thread-1 | OSGiDOMSchemaService             | 242 - org.opendaylight.mdsal.dom-schema-osgi - 8.0.5 | DOM Schema services deactivated
2021-09-20T16:13:12,695 | INFO  | opendaylight-cluster-data-akka.actor.default-dispatcher-4 | RepointableActorRef              | 205 - org.opendaylight.controller.repackaged-akka - 4.0.3 | Message [org.opendaylight.controller.cluster.common.actor.ExecuteInSelfMessage] to Actor[akka://opendaylight-cluster-data/user/
shardmanager-config#795186601] was not delivered. [6] dead letters encountered. If this is not an expected behavior then Actor[akka://opendaylight-cluster-data/user/shardmanager-config#795186601] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-
letters' and 'akka.log-dead-letters-during-shutdown'.
2021-09-20T16:13:12,696 | INFO  | opendaylight-cluster-data-akka.actor.default-dispatcher-34 | RepointableActorRef              | 205 - org.opendaylight.controller.repackaged-akka - 4.0.3 | Message [org.opendaylight.controller.cluster.databroker.actors.dds.AbstractDataStoreClientBehavior$$Lambda$2370/0x00000007c166
ec40] to Actor[akka://opendaylight-cluster-data/user/$a#2016588676] was not delivered. [7] dead letters encountered. If this is not an expected behavior then Actor[akka://opendaylight-cluster-data/user/$a#2016588676] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration set
tings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.

The key issue seems to be that we are not cleanly handing of BindingRuntimeContexts – we should be just adopting gen=2 and not deactivate services. The 'reason 5' deactivations are okay, as those are explicit disposals due to the old runtime context going away.

OpenDaylight JIRA

[MDSAL-689] Multiple issues during global context update