Operating System: All
Platform: All

Issue is netvirt-sfc does not load correctly in karaf when using this featuresBoot:
featuresBoot=config,standard,region,package,kar,ssh,management,odl-ovsdb-sfc-rest

Karaf logs:
https://www.dropbox.com/s/x6fw44wkmy2gesz/karaf_log.zip?dl=0

Using this featuresBoot seems to work:
featuresBoot=config,standard,region,package,kar,ssh,management,odl-sfc-ui,odl-sfc-sb-rest,odl-sfc-ovs,odl-sfc-netconf,odl-sfclisp,odl-sfcofl2,odl-ovsdb-openstack

Hi Sam,

So everything on net-virt-sfc side of things looks ok and that indicates a timing issue in ODL’s configuration subsystem. It looks like the check for required-capabilities and config subsystem itself are not in 100% sync. Not sure why the issue popped up now, but we need to find out. So here’s what we need to do:

1. Report a high priority bug to bugs.opendaylight.org for config subsystem. Attaching the last few posts from this conversation to provide enough details for further analysis and fix. If you could do that Sam, it would be great.

2. Try to find a workaround in the meantime. What you could do to minimize the race condition on your side is: Take out the config file from your odl-ovsdb-sfc feature into a dedicated feature. And either install that new feature manually after odl-ovsdb-sfc feature was processed, or just add the new feature as another boot feature, to try to increase the delay between installing the bundle and its configuration. I know this is not ideal, but it might help until the underlying issue gets resolved.

Maros

From: Sam Hague shague@redhat.com
Sent: Monday, April 04, 2016 11:28 PM
To: Maros Marsalek -X (mmarsale - PANTHEON TECHNOLOGIES at Cisco) <mmarsale@cisco.com>
Cc: Tim Rozet <trozet@redhat.com>; Edward Warnicke <hagbard@gmail.com>; Robert Varga <nite@hq.sk>; Stephen Kitt <skitt@redhat.com>

Subject: Re: Feature fails to load with config namespace NullPointerException

Maros,

thanks for the info. Comments inline.

Thanks, Sam

On Fri, Apr 1, 2016 at 3:23 AM, Maros Marsalek -X (mmarsale - PANTHEON TECHNOLOGIES at Cisco) <mmarsale@cisco.com> wrote:

Hey Tim, Sam

This would be my initial analysis:

The error is trying to say that namespace: urn:opendaylight:params:xml:ns:yang:netvirt:sfc that’s present in config file: openstack.net-virt-sfc.xml could not be found in any YANG model loaded by ODL. And there are a couple possible issues:

1. Namespace does not exist and YANG models describing the configuration have different namespace. Please check YANG models if the namespace is OK.

Looks good but could take a quick look at the files: [1] and [2]. This has been working for months and only recently started failing.
[1] https://github.com/opendaylight/ovsdb/blob/master/openstack/net-virt-sfc/impl/src/main/yang/netvirt-sfc.yang
[2] https://github.com/opendaylight/ovsdb/blob/master/openstack/net-virt-sfc/impl/src/main/config/default-config.xml

2. Bundle with YANG model with required namespace was not loaded successfully by ODL. Please check loaded bundles using bundle:list command in karaf CLI to check whether the bundle containing required YANG file is present there and marked as ACTIVE.

The bundle with the namespace is showing as ACTIVE from the bundle:list

3. Bundle with YANG model with required namespace was not loaded before the snapshot was applied == timing issue. Please check the <required-capabilities> section in your config file whether it contains an entry for required YANG model. If not, just add it there to make sure config snapshot gets pushed at the right time.

[2] above shows the required-caps for the model.

I would think there is a timing issue since this only fails in certain setups. In all cases we simply start with the feature in the bootFeature and hit the namespace issue. But it does work in other cases. I have tried to reorder some of the features but that doesn't help either.

What are some other things we can do to debug the issue?

4. There is an unknown bug.

We need to go over the above possibilities to find the real issue here.

Maros

----Original Message----
From: Tim Rozet trozet@redhat.com
Sent: Friday, April 01, 2016 3:13 AM
To: Sam Hague <shague@redhat.com>
Cc: Edward Warnicke <hagbard@gmail.com>; Robert Varga <nite@hq.sk>; Maros Marsalek -X (mmarsale - PANTHEON TECHNOLOGIES at Cisco) <mmarsale@cisco.com>; Stephen Kitt <skitt@redhat.com>
Subject: Re: Feature fails to load with config namespace NullPointerException

Also, I'm not sure you want to go through this much trouble, but I think it's the best shot at reproducing (I can reproduce everytime doing this):
1. On a CentOS 7 machine. Download these:
http://artifacts.opnfv.org/apex/brahmaputra/opnfv-apex-common-2.1-brahmaputra20160329.noarch.rpm
http://artifacts.opnfv.org/apex/brahmaputra/opnfv-apex-opendaylight-sfc-2.1-brahmaputra20160329.noarch.rpm
http://artifacts.opnfv.org/apex/brahmaputra/opnfv-apex-common-2.1-brahmaputra20160329.noarch.rpm

2. yum -y install opnfv-apex-opendaylight-sfc-2.1-brahmaputra20160329.noarch.rpm opnfv-apex-undercloud-2.1-brahmaputra20160329.noarch.rpm opnfv-apex-common-2.1-brahmaputra20160329.noarch.rpm

3. sudo opnfv-deploy -v -n /etc/opnfv-apex/network_settings.yaml -d /etc/opnfv-apex/os-odl_l2-sfc-noha.yaml
This will install the SFC OPNFV deployment, takes about 30 minutes.

4. After install completes. Login to the undercloud (installer VM):
ssh root@$(arp -a | grep $(virsh domiflist instack | grep default | awk '

') | grep -Eo "[0-9]\.[0-9]\.[0-9]\.[0-9]") su - stack source stackrc nova list

5. You should see 2 overcloud nodes, login to the controller:
-----------------------------------------------------------------------------------------------------------+

-----------------------------------------------------------------------------------------------------------+

-----------------------------------------------------------------------------------------------------------+

ssh heat-admin@192.0.2.3
sudo -i

6. OpenDaylight will be in /opt/opendaylight and will already be started. You should be able to find the error in karaf.log.

The only difference between the controller VM and regular centos7 is we are using a different kernel (3.13.7) because it supports the OVS+NSH kernel module. I've tried killing all of the OpenStack processes thinking it was a CPU issue, and that didn't help (still see the error). I also tried shutting down OVS on both nodes so that they wouldn't be trying to connect to ODL on bootup, that didn't seem to help either.

Tim Rozet
Red Hat SDN Team

----- Original Message -----
From: "Sam Hague" <shague@redhat.com>
To: "Edward Warnicke" <hagbard@gmail.com>
Cc: "Robert Varga" <nite@hq.sk>, "Maros Marsalek -X (mmarsale - PANTHEON TECHNOLOGIES at Cisco)" <mmarsale@cisco.com>, "Stephen Kitt" <skitt@redhat.com>, "Tim Rozet" <trozet@redhat.com>
Sent: Thursday, March 31, 2016 8:47:33 PM
Subject: Re: Feature fails to load with config namespace NullPointerException

Ed,

can you try using the rpms? I think the problem happens most often with the rpm install.

Just yum install the rpm on a centos/fedora/rhel setup. This will create a systemctl service that you can then start: sudo systemctl start opendaylight. Edit the featureCfg to include odl-ovsdb-sfc-rest in the bootFeatures. It is installed in /opt.

Thanks, Sam

[1] beryllium:
http://cbs.centos.org/repos/nfv7-opendaylight-40-release/x86_64/os/Packages/

[2] beryllium_sr1:
http://cbs.centos.org/repos/nfv7-opendaylight-41-release/x86_64/os/Packages/

On Thu, Mar 31, 2016 at 5:59 PM, Edward Warnicke <hagbard@gmail.com> wrote:

> And just repeated with:
>
> java -version
>
> java version "1.8.0_77"
>
> Java(TM) SE Runtime Environment (build 1.8.0_77-b03)
>
> Java HotSpot(TM) 64-Bit Server VM (build 25.77-b03, mixed mode)
>
>
> Still can't reproduce
>
>
> Ed
>
> On Thu, Mar 31, 2016 at 4:40 PM, Edward Warnicke <hagbard@gmail.com>
> wrote:
>
>> And downloaded
>>
>>
>> https://nexus.opendaylight.org/content/repositories/opendaylight.snap
>> shot/org/opendaylight/integration/distribution-karaf/0.5.0-SNAPSHOT/d
>> istribution-karaf-0.5.0-20160331.205306-1450.zip
>>
>> Running that using
>>
>> featuresBoot=config,standard,region,package,kar,ssh,
>> management,odl-ovsdb-sfc-rest
>>
>> with
>>
>> java version "1.8.0_60"
>>
>> Java(TM) SE Runtime Environment (build 1.8.0_60-b27)
>>
>> Java HotSpot(TM) 64-Bit Server VM (build 25.60-b23, mixed mode)
>>
>> also appears to work.
>>
>>
>>
>> On Thu, Mar 31, 2016 at 4:33 PM, Edward Warnicke <hagbard@gmail.com>
>> wrote:
>>
>>> OK... preliminary report...
>>>
>>> java -version
>>>
>>> java version "1.8.0_60"
>>>
>>> Java(TM) SE Runtime Environment (build 1.8.0_60-b27)
>>>
>>> Java HotSpot(TM) 64-Bit Server VM (build 25.60-b23, mixed mode)
>>>
>>>
>>> Using
>>>
>>>
>>> featuresBoot=config,standard,region,package,kar,ssh,management,odl-o
>>> vsdb-sfc-rest
>>>
>>> In ovsdb/openstack/net-virt-sfc/karaf (I was already running there,
>>> so it was the quickest thing to try)
>>>
>>> Works (no error like what you are seeing).
>>>
>>> On Thu, Mar 31, 2016 at 4:30 PM, Edward Warnicke <hagbard@gmail.com>
>>> wrote:
>>>
>>>> Yeah... Tim's comment in the bug about features was very useful.
>>>>
>>>> Ed
>>>>
>>>> On Thu, Mar 31, 2016 at 4:28 PM, Sam Hague <shague@redhat.com> wrote:
>>>>
>>>>> Ed,
>>>>>
>>>>> no, just use the official beryllium or sr1 release and add
>>>>> odl-ovsdb-sfc-rest to the featuresBoot. Fire it up and see if you
>>>>> hit the error.
>>>>>
>>>>> Only thinkg we can think of right now is maybe a java version
>>>>> difference. I am using 1.8.0.25 and works fine. Have vms with
>>>>> 1.8.0.65 and works fine. Tim has 1.8.0.77 and it is blowing up.
>>>>> Sometimes Tim can get it to work by reordering the dependent
>>>>> features and loading them first, but it is not reliable - can't
>>>>> pin down the issue. Also seems to only happen when using featuresBoot (I think).
>>>>>
>>>>> This also is using a rpm packaged ODL. Tim can forward the link
>>>>> for the download for that. But that RPM also works fine for me.
>>>>>
>>>>> Thanks, Sam
>>>>>
>>>>> On Thu, Mar 31, 2016 at 5:15 PM, Edward Warnicke
>>>>> <hagbard@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> So... can I reproduce this buy building and running:
>>>>>>
>>>>>> ovsdb/openstack/net-virt-sfc/karaf
>>>>>>
>>>>>> (is seeking the fast lazy path to reproduction ).
>>>>>>
>>>>>>
>>>>>> Ed
>>>>>>
>>>>>> On Thu, Mar 31, 2016 at 4:00 PM, Sam Hague <shague@redhat.com> wrote:
>>>>>>
>>>>>>> including Ed on thread.
>>>>>>>
>>>>>>> On Thu, Mar 31, 2016 at 4:59 PM, Sam Hague <shague@redhat.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Robert, Maros,
>>>>>>>>
>>>>>>>> could you take a look at this bug [1] and provide some guidance
>>>>>>>> on how to debug the issue?
>>>>>>>>
>>>>>>>> This final feature fails to load. What is difficult is this was
>>>>>>>> working fine for quite a while. Recently, and only on certain
>>>>>>>> setups does it fail with the exception. Many different setups
>>>>>>>> it works fine - different hosts, java versions etc - all works
>>>>>>>> good. But in a few setups it always fails.
>>>>>>>>
>>>>>>>> The bug below has a large karaf log with more debugging enabled.
>>>>>>>> The final feature odl-ovsdb-sfu-rest looks like everything it
>>>>>>>> needs has loaded so we can't see what is missing.
>>>>>>>>
>>>>>>>> Any help in how to debug the issue would be great.
>>>>>>>>
>>>>>>>> Thanks, Sam
>>>>>>>>
>>>>>>>> [1] https://bugs.opendaylight.org/show_bug.cgi?id=5585
>>>>>>>>
>>>>>>>> 2016-03-04 16:38:30,860 | ERROR | config-pusher |
>>>>>>>> ConfigPusherImpl | 92 -
>>>>>>>> org.opendaylight.controller.config-persister-impl -
>>>>>>>> 0.4.0.Beryllium | Failed to apply configurat:ion snapshot:
>>>>>>>> openstack.net-virt-sfc.xml(odl-ovsdb-sfc,odl-ovsdb-sfc)
>>>>>>>> java.lang.NullPointerException: Namespace
>>>>>>>> urn:opendaylight:params:xml:ns:yang:netvirt:sfc, defined in:
>>>>>>>> module netvirt-sfc-default of type netvirt-sfc not found, available namespaces:
>>>>>>>> [urn:opendaylight:params:xml:ns:yang:topology-manager:impl,
>>>>>>>> urn:opendaylight:params:xml:ns:yang:neutron:transcriber:impl,
>>>>>>>> urn:opendaylight:params:xml:ns:yang:library:impl,
>>>>>>>> urn:opendaylight:params:xml:ns:yang:netvirt:providers:impl,
>>>>>>>> urn:opendaylight:params:xml:ns:yang:controller:config:actor-sys
>>>>>>>> tem-provider:impl,
>>>>>>>> urn:opendaylight:params:xml:ns:yang:openflow:common:config:impl
>>>>>>>> ,
>>>>>>>> urn:opendaylight:params:xml:ns:yang:openflowplugin:ofjava:nx:ap
>>>>>>>> i:config, urn:opendaylight:params:xml:ns:yang:netvirt:impl,
>>>>>>>> urn:opendaylight:params:xml:ns:yang:openflow:applications:lldp-
>>>>>>>> speaker,

>>>>>>>> urn:opendaylight:params:xml:ns:yang:controller:config:cluster-a
>>>>>>>> dmin-provider,
>>>>>>>> urn:opendaylight:params:xml:ns:yang:controller:netty:eventexecu
>>>>>>>> tor,
>>>>>>>> urn:opendaylight:params:xml:ns:yang:controller:threadpool:impl:
>>>>>>>> fixed,
>>>>>>>> urn:opendaylight:params:xml:ns:yang:controller:sal:restconf:ser
>>>>>>>> vice,
>>>>>>>> urn:opendaylight:params:xml:ns:yang:controller:md:sal:dom:impl,
>>>>>>>> urn:opendaylight:params:xml:ns:yang:controller:inmemory-datasto
>>>>>>>> re-provider,
>>>>>>>> urn:opendaylight:params:xml:ns:yang:controller:netty:timer,
>>>>>>>> urn:opendaylight:params:xml:ns:yang:controller:config:distribut
>>>>>>>> ed-datastore-provider,
>>>>>>>> urn:opendaylight:params:xml:ns:yang:openflowplugin:ofjava:nx:co
>>>>>>>> nfig,
>>>>>>>> urn:opendaylight:params:xml:ns:yang:openflow:switch:connection:
>>>>>>>> provider:impl,
>>>>>>>> urn:opendaylight:params:xml:ns:yang:controller:threadpool:impl,
>>>>>>>> urn:opendaylight:params:xml:ns:yang:controller:config:distribut
>>>>>>>> ed-entity-ownership-service,
>>>>>>>> urn:opendaylight:params:xml:ns:yang:topology-lldp-discovery:imp
>>>>>>>> l,
>>>>>>>> urn:opendaylight:params:xml:ns:yang:controller:md:sal:rest:conn
>>>>>>>> ector,
>>>>>>>> urn:opendaylight:params:xml:ns:yang:controller:md:sal:dom:pingp
>>>>>>>> ong,
>>>>>>>> urn:opendaylight:params:xml:ns:yang:openflowplugin:app:statisti
>>>>>>>> cs-manager,
>>>>>>>> urn:opendaylight:params:xml:ns:yang:controller:md:sal:binding:i
>>>>>>>> mpl,
>>>>>>>> urn:opendaylight:params:xml:ns:yang:controller:config:sfc-ovs:i
>>>>>>>> mpl,
>>>>>>>> urn:opendaylight:params:xml:ns:yang:openflow:applications:of-sw
>>>>>>>> itch-config-pusher,
>>>>>>>> urn:opendaylight:params:xml:ns:yang:inventory-manager:impl,
>>>>>>>> http://netconfcentral.org/ns/XSQL,
>>>>>>>> urn:opendaylight:params:xml:ns:yang:openflowplugin:app:forwardi
>>>>>>>> ngrules-manager,
>>>>>>>> config:aaa:authn:h2:store,
>>>>>>>> urn:opendaylight:params:xml:ns:yang:southbound:impl,
>>>>>>>> urn:opendaylight:params:xml:ns:yang:controller:threadpool:impl:
>>>>>>>> flexible,
>>>>>>>> urn:opendaylight:params:xml:ns:yang:controller:config:sfc-provi
>>>>>>>> der:impl,
>>>>>>>> urn:opendaylight:params:xml:ns:yang:openflowplugin:nx:config:im
>>>>>>>> pl,
>>>>>>>> urn:opendaylight:params:xml:ns:yang:controller:config:sfc-l2ren
>>>>>>>> derer:impl,
>>>>>>>> urn:opendaylight:params:xml:ns:yang:controller:netty:threadgrou
>>>>>>>> p,
>>>>>>>> urn:opendaylight:params:xml:ns:yang:controller:config:concurren
>>>>>>>> t-data-broker,

>>>>>>>> urn:opendaylight:params:xml:ns:yang:controller:shutdown:impl,
>>>>>>>> urn:opendaylight:params:xml:ns:yang:controller:config:remote-rp
>>>>>>>> c-connector,
>>>>>>>> urn:opendaylight:params:xml:ns:yang:controller:threadpool:impl:
>>>>>>>> scheduled,
>>>>>>>> config:aaa:authn:idmlight]
>>>>>>>> at
>>>>>>>> com.google.common.base.Preconditions.checkNotNull(Preconditions.java:250)[39:com.google.guava:18.0.0]
>>>>>>>> at
>>>>>>>> org.opendaylight.controller.config.facade.xml.mapping.config.Config.getModuleMapping(Config.java:281)[90:org.opendaylight.controller.config-manager-facade-xml:0.4.0.Beryllium]
>>>>>>>> at
>>>>>>>> org.opendaylight.controller.config.facade.xml.mapping.config.Config.resolveModule(Config.java:216)[90:org.opendaylight.controller.config-manager-facade-xml:0.4.0.Beryllium]
>>>>>>>> at
>>>>>>>> org.opendaylight.controller.config.facade.xml.mapping.config.Config.fromXmlModulesMap(Config.java:181)[90:org.opendaylight.controller.config-manager-facade-xml:0.4.0.Beryllium]
>>>>>>>> at
>>>>>>>> org.opendaylight.controller.config.facade.xml.ConfigExecution.getModulesDefinition(ConfigExecution.java:58)[90:org.opendaylight.controller.config-manager-facade-xml:0.4.0.Beryllium]
>>>>>>>> at
>>>>>>>> org.opendaylight.controller.config.facade.xml.ConfigSubsystemFacade.handleMisssingInstancesOnTransaction(ConfigSubsystemFacade.java:246)[90:org.opendaylight.controller.config-manager-facade-xml:0.4.0.Beryllium]
>>>>>>>> at
>>>>>>>> org.opendaylight.controller.config.facade.xml.ConfigSubsystemFacade.test(ConfigSubsystemFacade.java:163)[90:org.opendaylight.controller.config-manager-facade-xml:0.4.0.Beryllium]
>>>>>>>> at
>>>>>>>> org.opendaylight.controller.config.facade.xml.ConfigSubsystemFacade.executeTests(ConfigSubsystemFacade.java:149)[90:org.opendaylight.controller.config-manager-facade-xml:0.4.0.Beryllium]
>>>>>>>> at
>>>>>>>> org.opendaylight.controller.config.facade.xml.ConfigSubsystemFacade.executeConfigExecution(ConfigSubsystemFacade.java:123)[90:org.opendaylight.controller.config-manager-facade-xml:0.4.0.Beryllium]
>>>>>>>> at
>>>>>>>> org.opendaylight.controller.config.persist.impl.ConfigPusherImpl.executeWithMissingModuleFactoryRetries(ConfigPusherImpl.java:329)[92:org.opendaylight.controller.config-persister-impl:0.4.0.Beryllium]
>>>>>>>> at
>>>>>>>> org.opendaylight.controller.config.persist.impl.ConfigPusherImpl.pushConfig(ConfigPusherImpl.java:305)[92:org.opendaylight.controller.config-persister-impl:0.4.0.Beryllium]
>>>>>>>> at
>>>>>>>> org.opendaylight.controller.config.persist.impl.ConfigPusherImpl.pushConfigWithConflictingVersionRetries(ConfigPusherImpl.java:164)[92:org.opendaylight.controller.config-persister-impl:0.4.0.Beryllium]
>>>>>>>> at
>>>>>>>> org.opendaylight.controller.config.persist.impl.ConfigPusherImpl.internalPushConfigs(ConfigPusherImpl.java:129)[92:org.opendaylight.controller.config-persister-impl:0.4.0.Beryllium]
>>>>>>>> at
>>>>>>>> org.opendaylight.controller.config.persist.impl.ConfigPusherImpl.processSingle(ConfigPusherImpl.java:83)[92:org.opendaylight.controller.config-persister-impl:0.4.0.Beryllium]
>>>>>>>> at
>>>>>>>> org.opendaylight.controller.config.persist.impl.ConfigPusherImpl.process(ConfigPusherImpl.java:74)[92:org.opendaylight.controller.config-persister-impl:0.4.0.Beryllium]
>>>>>>>> at
>>>>>>>> org.opendaylight.controller.config.persist.impl.osgi.ConfigPersisterActivator$2.run(ConfigPersisterActivator.java:131)[92:org.opendaylight.controller.config-persister-impl:0.4.0.Beryllium]
>>>>>>>> at java.lang.Thread.run(Thread.java:745)[:1.8.0_71]
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>

Jenkins job showing the error: https://jenkins.opendaylight.org/releng/view/ovsdb/job/ovsdb-daily-full-integration-boron/86/OVS_VERSION=2.4.0,jdk=openjdk8,nodes=dynamic_docker/console

I ran various tests with modify the boot features, capabilities and features for odl-ovsdb-sfc. If I removed the odl-sfcofl2 and odl-sfc-provider features from the odl-ovsdb-sfc feature then the config snapshot would load, but then there would be an error because the netvirt-sfc code relies on models provided by odl-sfc-provider.

I also tried to add a req cap for the sfc-provider and it had no impact on the model failure.

I also tried to include on the odl-sfc-model feature with odl-ovsdb-sfc and it caused the config namespace error.

I also put the odl-sfc* features in the bootFeatures and had no success.

So it seems that there is something in the odl-sfc features conflicting with the netvirt-sfc config loading.

Attachment karaf.log.bz2 has been added with description: Failing config snapshot

(In reply to Sam Hague from comment #5)
> Created attachment 925 [details]
> Failing config snapshot

This is a karaf.log with config st to TRACE level.

Sam,

Are you still seeing this?

Are the reproduction steps still the same:

1.) launch ODL using featuresBoot=featuresBoot=config,standard,region,package,kar,ssh,management,odl-ovsdb-sfc-rest
2.) install netvirt-sfc
3.) Look for the Failed to apply configuration snapshot: openstack.net-virt-sfc.xml exception.

Colin,
I am moving this to net-virt project, because net-virt-sfc is moved to blueprint now, so we need to reinvestigate this to make sure this issue still exist or not.