JankiChhatbar and k.faseela, I'm not sure I really understand this issue, but is this something I can help with?

dfarrell07 i am not sure about the changes involved, but somewhere there is an upgrade-in-progress flag setting, which is currently pointing to a rest URL in genius, and that needs to be modified to use the migrated URL from serviceutils project

Okay, I think I tracked down what needs to change:

https://github.com/openstack/tripleo-heat-templates/blob/master/docker/services/opendaylight-api.yaml

Those are tasks in THT that modify the flag as part of an upgrade process. For sure the REST call needs to change from Genius to Serviceutils. Maybe also something about the match conditions when editing the file to change the var, but need to verify.

Just documenting these here so I don't have to include the very long links in commit messages.

This is the old Genius upgradeInProgress REST API definition:

https://git.opendaylight.org/gerrit/gitweb?p=genius.git;a=blob;f=mdsalutil/mdsalutil-api/src/main/yang/mdsalutil.yang;h=e367a375623627e2aeae516c948390e857eac50f;hb=HEAD

This is the new Serviceutils upgradeInProgress REST API definition:

https://git.opendaylight.org/gerrit/gitweb?p=serviceutils.git;a=blob;f=upgrade/src/main/yang/upgrade-config.yang;h=553ef966855c0185c3b49857a7ee550c180c6a4b;hb=HEAD

Comparing the two, you can see what needs to change.

Patch: https://review.openstack.org/#/c/621694/

I am constantly seeing this error in karaf logs for Oxygen. Is this the side effect of this?

2018-12-04T08:13:42,175 | ERROR | ConfigFeatureListener - ConfigPusher | AbstractFeatureWrapper           | 52 - config-persister-feature-adapter - 0.8.4.redhat-7 | Could not parse XML file /opt/opendaylight/etc/opendaylight/datastore/initial/config/serviceutils-upgrade-config.xml
java.io.FileNotFoundException: /opt/opendaylight/etc/opendaylight/datastore/initial/config/serviceutils-upgrade-config.xml (No such file or directory)
        at java.io.FileInputStream.open0(Native Method) ~[?:?]
        at java.io.FileInputStream.open(FileInputStream.java:195) [?:?]
        at java.io.FileInputStream.<init>(FileInputStream.java:138) [?:?]
        at org.opendaylight.controller.configpusherfeature.internal.AbstractFeatureWrapper.isConfigSnapshot(AbstractFeatureWrapper.java:116) [52:config-persister-feature-adapter:0.8.4.redhat-7]
        at org.opendaylight.controller.configpusherfeature.internal.AbstractFeatureWrapper.getFeatureConfigSnapshotHolders(AbstractFeatureWrapper.java:75) [52:config-persister-feature-adapter:0.8.4.redhat-7]
        at org.opendaylight.controller.configpusherfeature.internal.ChildAwareFeatureWrapper.getFeatureConfigSnapshotHolders(ChildAwareFeatureWrapper.java:89) [52:config-persister-feature-adapter:0.8.4.redhat-7]
        at org.opendaylight.controller.configpusherfeature.internal.ChildAwareFeatureWrapper.getFeatureConfigSnapshotHolders(ChildAwareFeatureWrapper.java:81) [52:config-persister-feature-adapter:0.8.4.redhat-7]
        at org.opendaylight.controller.configpusherfeature.internal.FeatureConfigPusher.pushConfig(FeatureConfigPusher.java:101) [52:config-persister-feature-adapter:0.8.4.redhat-7]
        at org.opendaylight.controller.configpusherfeature.internal.FeatureConfigPusher.pushConfigs(FeatureConfigPusher.java:69) [52:config-persister-feature-adapter:0.8.4.redhat-7]
        at org.opendaylight.controller.configpusherfeature.internal.ConfigPushingRunnable.processFeatureEvent(ConfigPushingRunnable.java:70) [52:config-persister-feature-adapter:0.8.4.redhat-7]
        at org.opendaylight.controller.configpusherfeature.internal.ConfigPushingRunnable.run(ConfigPushingRunnable.java:52) [52:config-persister-feature-adapter:0.8.4.redhat-7]
        at java.lang.Thread.run(Thread.java:748) [?:?]

JankiChhatbar That's initial config file relevant to this, yes. It's not clear to me that it's related to this work exactly though.

This same Serviceutils upgrade config is apparently backported all the way back to stable/oxygen:

https://git.opendaylight.org/gerrit/gitweb?p=serviceutils.git;a=blob;f=upgrade/src/main/yang/upgrade-config.yang;h=553ef966855c0185c3b49857a7ee550c180c6a4b;hb=refs/heads/stable/oxygen

It seems to me like something in ODL is expecting that config file and logging the error when it's missing. That doesn't seem right, as the initial config should be optional.

JankiChhatbar and dfarrell07 the error above means that /opt/opendaylight/etc/opendaylight/datastore/initial/config/serviceutils-upgrade-config.xml is missing ... that is https://github.com/opendaylight/serviceutils/blob/stable/oxygen/upgrade/src/main/resources/initial/serviceutils-upgrade-config.xml which yes is the "initial config file relevant to this" (which you can then override by RESTCONF). That file should be there... it's weird that you "lost" it... https://github.com/opendaylight/serviceutils/blob/stable/oxygen/features/odl-serviceutils-tools/pom.xml should have installed this. (We could try to loally build an Oxygen netvirt/karaf to "prove" it's normally there?) Because https://github.com/opendaylight/serviceutils/blob/stable/oxygen/upgrade/src/main/yang/upgrade-config.yang#L21 defaults to false anyway (it's not nice how we duplicate stuff...), it's probably not breaking anything, but this error will likely lead to confusion, so would be good to clarify how we "loose" the serviceutils-upgrade-config.xml and make sure it's there.

Atleast I do see this file in the target folder :

C:\netvirt\karaf\target\assembly\etc\opendaylight\datastore\initial\config>ls
aaa-cert-config.xml default-openflow-connection-config.xml genius-itm-config.xml legacy-openflow-connection-config.xml
aaa-encrypt-service-config.xml genius-ifm-config.xml genius-mdsalutil-config.xml serviceutils-upgrade-config.xml

C:\Users\efaseel\repo\odl_upstream\netvirt\karaf\target\assembly\etc\opendaylight\datastore\initial\config>

Just to be extra sure, I just tested that in a locally freshly built Oxygen netvirt Karaf, after feature:install odl-netvirt-openstack it's OK, and indeed I do have /home/vorburger/dev/ODL/git/netvirt/karaf/target/assembly/etc/opendaylight/datastore/initial/config/serviceutils-upgrade-config.xml present there (on Oxygen already, not just master), so it would seem like this somehow gets lost / removed / deleted by something in INTPACK somewhere...

> so it would seem like this somehow gets lost / removed / deleted by something in INTPACK somewhere...

That would be very surprising to me, but I can't rule it out without some digging.

JankiChhatbar - This is just a normal, out of the box deployment where you're seeing this error?

dfarrell07 Yes. But this downstream RPM deployment so probably all patches are not available downstream yet. Would any of you mind confirming if they are.

Actually, this sounds a lot like it could be another problem caused by https://bugs.launchpad.net/tripleo/+bug/1805859.

I have that file too. Looks like the error is well before the file was generated.

2018-12-06T10:30:36,616 | ERROR | ConfigFeatureListener - ConfigPusher | AbstractFeatureWrapper           | 52 - config-persister-feature-adapter - 0.8.4.redhat-7 | Could not parse XML file /opt/opendaylight/etc/opendaylight/datastore/initial/config/serviceutils-upgrade-config.xml
java.io.FileNotFoundException: /opt/opendaylight/etc/opendaylight/datastore/initial/config/serviceutils-upgrade-config.xml (No such file or directory)

$ ls -l etc/opendaylight/datastore/initial/config/serviceutils-upgrade-config.xml
rw-rr-. 1 odl odl 130 Dec  6 14:27 etc/opendaylight/datastore/initial/config/serviceutils-upgrade-config.xml

JankiChhatbar now that's really curious... and you obviously were in /opt/opendaylight when you did that ls?

Not sure what's going on there. Perhaps some ordering thing, I suspect it wasn't there yet when it ran but was there when you checked... FYI it's the feature:install odl-netvirt-openstack that lays down that file there (it's not in the original ZIP and probably RPM), but that ConfigFeatureListener - ConfigPusher | AbstractFeatureWrapper | config-persister-feature-adapter thing should only run after that feature:install. Given how confusing this is, I would say ignore it for now, unless you are seeing a "real" problem.

Is there a patch for this, which I can take a look at?

Just an FYI that, we are going to merge the Genius patch soon.

Trying to remember the details for k.faseela, didn't this end up being part of https://bugzilla.redhat.com/show_bug.cgi?id=1623123 and fixed in OpenStack?

dfarrell07 : Thanks, if so can you close this, so that I can just focus on the CSIT side changes and genius side changes alone?

dfarrell07 : Do you think this patch is still needed, and it needs to be merged?

https://review.openstack.org/#/c/621694/

Genius side changes are merged, just FYI