Operating System: All
Platform: All

Unfortunately this is not this easy. Models can be introduced into the system pretty much at any time and there is inherently not an indication when the operator deems the system to be configured. With Karaf, it is normal to install features incrementally, which means that while there are interim steps, where the system is consistent, during installs it is perfectly possible for two bundles to see two versions of the SchemaContext, which obviously can lead to a mid-air collision.

Incrementally adding new models to a running controller is fine, the CDS handles that.

I'm referring to startup with existing models that were installed prior to a restart. If there's persisted data for that model in a shard and if the schema is not yet present in the SchemaContext, i.e. the bundle that provides the model/schema hasn't been activated yet, then persistence recovery will fail. This was seen some times - it depends on bundle ordering and threading so it may work fine 9 times out of 10. When a Shard actor is created, akka first recovers it from persistence before "activating" it, i.e. before it allows the actor to process normal messages.

To alleviate persistence recovery failures, we had to delay shard creation in the ShardManager until schemas have been loaded. However it's not deterministic, i.e. we can't tell which models/schemas can potentially exist in a shard. So as schemas are loaded, we persist the current list of schema module names in the ShardManager actor. The ShardManager also compares the current list against the list that was recovered from persistence on startup. Only when the current list of loaded schemas contains all the entries in the recovered list do we deem it safe to start creating shards. The only hole in this mechanism is if a model's schema that was previously persisted is removed on restart, e.g. if the controller is stopped for upgrade and a feature is removed from featuresBoot or a model is removed from a bundle (obsoleted) and the data/cache directory is cleaned so features are re-installed. In that case, the ShardManager would wait forever for the model's schema and wouldn't create any shards.

This mechanism and potential complications can be removed if all previously existing models were loaded by md-sal/config system prior to CDS startup. Changing ModuleInfoBackedBundleTracker to listen for RESOLVED bundles would achieve that.

When adding a new model dynamically vis feature:install, there's no problem b/c there's no persisted data yet plus the target shard has already started up.

Fixed by https://git.opendaylight.org/gerrit/#/c/20908/