Operating System: All
Platform: All

Attachment odl1_karaf.log.tar.xz has been added with description: karaf.log

2017-07-11 01:32:12,679 | ERROR | eChangeHandler-1 | AsyncDataTreeChangeListenerBase | 238 - org.opendaylight.genius.mdsalutil-api - 0.3.0.SNAPSHOT | Thread terminated due to uncaught exception: AsyncDataTreeChangeListenerBase-DataTreeChangeHandler-1
java.lang.NullPointerException
at org.opendaylight.netvirt.aclservice.StatefulIngressAclServiceImpl.programConntrackRecircRules(StatefulIngressAclServiceImpl.java:140)[292:org.opendaylight.netvirt.aclservice-impl:0.5.0.SNAPSHOT]
at org.opendaylight.netvirt.aclservice.StatefulIngressAclServiceImpl.programIngressAclFixedConntrackRule(StatefulIngressAclServiceImpl.java:161)[292:org.opendaylight.netvirt.aclservice-impl:0.5.0.SNAPSHOT]
at org.opendaylight.netvirt.aclservice.StatefulIngressAclServiceImpl.programSpecificFixedRules(StatefulIngressAclServiceImpl.java:80)[292:org.opendaylight.netvirt.aclservice-impl:0.5.0.SNAPSHOT]
at org.opendaylight.netvirt.aclservice.AbstractAclServiceImpl.programAclWithAllowedAddress(AbstractAclServiceImpl.java:229)[292:org.opendaylight.netvirt.aclservice-impl:0.5.0.SNAPSHOT]
at org.opendaylight.netvirt.aclservice.AbstractAclServiceImpl.removeAcl(AbstractAclServiceImpl.java:244)[292:org.opendaylight.netvirt.aclservice-impl:0.5.0.SNAPSHOT]
at org.opendaylight.netvirt.aclservice.AclServiceManagerImpl.notify(AclServiceManagerImpl.java:60)[292:org.opendaylight.netvirt.aclservice-impl:0.5.0.SNAPSHOT]
at org.opendaylight.netvirt.aclservice.listeners.AclInterfaceStateListener.remove(AclInterfaceStateListener.java:76)[292:org.opendaylight.netvirt.aclservice-impl:0.5.0.SNAPSHOT]
at org.opendaylight.netvirt.aclservice.listeners.AclInterfaceStateListener.remove(AclInterfaceStateListener.java:34)[292:org.opendaylight.netvirt.aclservice-impl:0.5.0.SNAPSHOT]
at org.opendaylight.genius.datastoreutils.AsyncDataTreeChangeListenerBase$DataTreeChangeHandler.run(AsyncDataTreeChangeListenerBase.java:150)[238:org.opendaylight.genius.mdsalutil-api:0.3.0.SNAPSHOT]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_131]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_131]
at java.lang.Thread.run(Thread.java:748)[:1.8.0_131]

2017-07-11 01:32:56,793 | ERROR | eChangeHandler-0 | AsyncDataTreeChangeListenerBase | 238 - org.opendaylight.genius.mdsalutil-api - 0.3.0.SNAPSHOT | Thread terminated due to uncaught exception: AsyncDataTreeChangeListenerBase-DataTreeChangeHandler-0
java.lang.NullPointerException
at org.opendaylight.netvirt.aclservice.AbstractAclServiceImpl.processInterfaceUpdate(AbstractAclServiceImpl.java:163)[292:org.opendaylight.netvirt.aclservice-impl:0.5.0.SNAPSHOT]
at org.opendaylight.netvirt.aclservice.AbstractAclServiceImpl.updateAcl(AbstractAclServiceImpl.java:142)[292:org.opendaylight.netvirt.aclservice-impl:0.5.0.SNAPSHOT]
at org.opendaylight.netvirt.aclservice.AclServiceManagerImpl.notify(AclServiceManagerImpl.java:57)[292:org.opendaylight.netvirt.aclservice-impl:0.5.0.SNAPSHOT]
at org.opendaylight.netvirt.aclservice.listeners.AclInterfaceListener.update(AclInterfaceListener.java:104)[292:org.opendaylight.netvirt.aclservice-impl:0.5.0.SNAPSHOT]
at org.opendaylight.netvirt.aclservice.listeners.AclInterfaceListener.update(AclInterfaceListener.java:33)[292:org.opendaylight.netvirt.aclservice-impl:0.5.0.SNAPSHOT]
at org.opendaylight.genius.datastoreutils.AsyncDataTreeChangeListenerBase$DataTreeChangeHandler.run(AsyncDataTreeChangeListenerBase.java:153)[238:org.opendaylight.genius.mdsalutil-api:0.3.0.SNAPSHOT]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_131]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_131]
at java.lang.Thread.run(Thread.java:748)[:1.8.0_131]

2017-07-11 01:32:12,677 | INFO | eChangeHandler-1 | AbstractIngressAclServiceImpl | 292 - org.opendaylight.netvirt.aclservice-impl - 0.5.0.SNAPSHOT | programFixedRules : removing default rules.
2017-07-11 01:32:12,679 | ERROR | eChangeHandler-1 | AsyncDataTreeChangeListenerBase | 238 - org.opendaylight.genius.mdsalutil-api - 0.3.0.SNAPSHOT | Thread terminated due to uncaught exception: AsyncDataTreeChangeListenerBase-DataTreeChangeHandler-1
java.lang.NullPointerException
at org.opendaylight.netvirt.aclservice.StatefulIngressAclServiceImpl.programConntrackRecircRules(StatefulIngressAclServiceImpl.java:140)[292:org.opendaylight.netvirt.aclservice-impl:0.5.0.SNAPSHOT]
at org.opendaylight.netvirt.aclservice.StatefulIngressAclServiceImpl.programIngressAclFixedConntrackRule(StatefulIngressAclServiceImpl.java:161)[292:org.opendaylight.netvirt.aclservice-impl:0.5.0.SNAPSHOT]
at org.opendaylight.netvirt.aclservice.StatefulIngressAclServiceImpl.programSpecificFixedRules(StatefulIngressAclServiceImpl.java:80)[292:org.opendaylight.netvirt.aclservice-impl:0.5.0.SNAPSHOT]
at org.opendaylight.netvirt.aclservice.AbstractAclServiceImpl.programAclWithAllowedAddress(AbstractAclServiceImpl.java:229)[292:org.opendaylight.netvirt.aclservice-impl:0.5.0.SNAPSHOT]
at org.opendaylight.netvirt.aclservice.AbstractAclServiceImpl.removeAcl(AbstractAclServiceImpl.java:244)[292:org.opendaylight.netvirt.aclservice-impl:0.5.0.SNAPSHOT]
at org.opendaylight.netvirt.aclservice.AclServiceManagerImpl.notify(AclServiceManagerImpl.java:60)[292:org.opendaylight.netvirt.aclservice-impl:0.5.0.SNAPSHOT]
at org.opendaylight.netvirt.aclservice.listeners.AclInterfaceStateListener.remove(AclInterfaceStateListener.java:76)[292:org.opendaylight.netvirt.aclservice-impl:0.5.0.SNAPSHOT]
at org.opendaylight.netvirt.aclservice.listeners.AclInterfaceStateListener.remove(AclInterfaceStateListener.java:34)[292:org.opendaylight.netvirt.aclservice-impl:0.5.0.SNAPSHOT]
at org.opendaylight.genius.datastoreutils.AsyncDataTreeChangeListenerBase$DataTreeChangeHandler.run(AsyncDataTreeChangeListenerBase.java:150)[238:org.opendaylight.genius.mdsalutil-api:0.3.0.SNAPSHOT]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_131]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_131]
at java.lang.Thread.run(Thread.java:748)[:1.8.0_131]

2017-07-10 20:32:36,279 | ERROR | pool-16-thread-1 | DOMNotificationRouterEvent | 144 - org.opendaylight.controller.sal-broker-impl - 1.5.1.SNAPSHOT | Delivery of notification org.opendaylight.controller.md.sal.binding.impl.LazySerializedDOMNotification@78021a0e caused an error in listener org.opendaylight.controller.md.sal.binding.impl.BindingDOMNotificationListenerAdapter@6a410f18
java.lang.NullPointerException
at org.opendaylight.netvirt.aclservice.listeners.AclVpnChangeListener.onAddInterfaceToDpnOnVpnEvent(AclVpnChangeListener.java:63)[343:org.opendaylight.netvirt.aclservice-impl:0.4.1.SNAPSHOT]
at org.opendaylight.yangtools.yang.binding.util.NotificationListenerInvoker.invokeNotification(NotificationListenerInvoker.java:117)[47:org.opendaylight.mdsal.yang-binding:0.10.1.SNAPSHOT]
at org.opendaylight.controller.md.sal.binding.impl.BindingDOMNotificationListenerAdapter.onNotification(BindingDOMNotificationListenerAdapter.java:44)[146:org.opendaylight.controller.sal-binding-broker-impl:1.5.1.SNAPSHOT]
at org.opendaylight.controller.md.sal.dom.broker.impl.DOMNotificationRouterEvent.deliverNotification(DOMNotificationRouterEvent.java:56)[144:org.opendaylight.controller.sal-broker-impl:1.5.1.SNAPSHOT]
at org.opendaylight.controller.md.sal.dom.broker.impl.DOMNotificationRouter$1.onEvent(DOMNotificationRouter.java:68)[144:org.opendaylight.controller.sal-broker-impl:1.5.1.SNAPSHOT]
at org.opendaylight.controller.md.sal.dom.broker.impl.DOMNotificationRouter$1.onEvent(DOMNotificationRouter.java:65)[144:org.opendaylight.controller.sal-broker-impl:1.5.1.SNAPSHOT]

17-07-10 20:40:26,747 | ERROR | ool-214-thread-5 | NaptEventHandler | 342 - org.opendaylight.netvirt.natservice-impl - 0.4.1.SNAPSHOT | NAT Service :Exception in NaptEventHandler.handleEvent() payload org.opendaylight.netvirt.natservice.internal.NAPTEntryEvent@4a83c660
java.lang.NullPointerException: the Function passed to Optional.transform() must not return null.
at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:226)[38:com.google.guava:18.0.0]
at com.google.common.base.Present.transform(Present.java:71)[38:com.google.guava:18.0.0]
at org.opendaylight.netvirt.natservice.internal.NatUtil.getExtGwMacAddFromRouterId(NatUtil.java:1525)[342:org.opendaylight.netvirt.natservice-impl:0.4.1.SNAPSHOT]
at org.opendaylight.netvirt.natservice.internal.NaptEventHandler.handleEvent(NaptEventHandler.java:193)[342:org.opendaylight.netvirt.natservice-impl:0.4.1.SNAPSHOT]
at org.opendaylight.netvirt.natservice.internal.NaptPacketInHandler$1.run(NaptPacketInHandler.java:139)[342:org.opendaylight.netvirt.natservice-impl:0.4.1.SNAPSHOT]

Added the last two exceptions from Vivek's email which is from a carbon run [1]. Logs are attached as odl_karaf.log.carbon.tar.xz. The first four exceptions were from the nitrogen run.

https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-newton-nodl-v2-upstream-stateful-carbon/807/odl1_karaf.log.gz

Attachment odl_karaf.log.carbon.tar.xz has been added with description: karaf.log.carbon

The exception in programConntrackRecircRules will be fixed by [1].
Trying to reproduce others.

[1]https://git.opendaylight.org/gerrit/#/c/60255/

(In reply to Sam Hague from comment #5)
> 17-07-10 20:40:26,747 | ERROR | ool-214-thread-5 | NaptEventHandler
> | 342 - org.opendaylight.netvirt.natservice-impl - 0.4.1.SNAPSHOT | NAT
> Service :Exception in NaptEventHandler.handleEvent() payload
> org.opendaylight.netvirt.natservice.internal.NAPTEntryEvent@4a83c660
> java.lang.NullPointerException: the Function passed to Optional.transform()
> must not return null.
> at
> com.google.common.base.Preconditions.checkNotNull(Preconditions.java:226)[38:
> com.google.guava:18.0.0]
> at
> com.google.common.base.Present.transform(Present.java:71)[38:com.google.
> guava:18.0.0]
> at
> org.opendaylight.netvirt.natservice.internal.NatUtil.
> getExtGwMacAddFromRouterId(NatUtil.java:1525)[342:org.opendaylight.netvirt.
> natservice-impl:0.4.1.SNAPSHOT]
> at
> org.opendaylight.netvirt.natservice.internal.NaptEventHandler.
> handleEvent(NaptEventHandler.java:193)[342:org.opendaylight.netvirt.
> natservice-impl:0.4.1.SNAPSHOT]
> at
> org.opendaylight.netvirt.natservice.internal.NaptPacketInHandler$1.
> run(NaptPacketInHandler.java:139)[342:org.opendaylight.netvirt.natservice-
> impl:0.4.1.SNAPSHOT]
Below is the exception from AclService.
java.lang.NullPointerException

at org.opendaylight.netvirt.aclservice.listeners.AclVpnChangeListener.onAddInterfaceToDpnOnVpnEvent(AclVpnChangeListener.java:63)[343:org.opendaylight.netvirt.aclservice-impl:0.4.1.SNAPSHOT]

at org.opendaylight.yangtools.yang.binding.util.NotificationListenerInvoker.invokeNotification(NotificationListenerInvoker.java:117)[47:org.opendaylight.mdsal.yang-binding:0.10.1.SNAPSHOT]

at org.opendaylight.controller.md.sal.binding.impl.BindingDOMNotificationListenerAdapter.onNotification(BindingDOMNotificationListenerAdapter.java:44)[146:org.opendaylight.controller.sal-binding-broker-impl:1.5.1.SNAPSHOT]

at org.opendaylight.controller.md.sal.dom.broker.impl.DOMNotificationRouterEvent.deliverNotification(DOMNotificationRouterEvent.java:56)[144:org.opendaylight.controller.sal-broker-impl:1.5.1.SNAPSHOT]

at org.opendaylight.controller.md.sal.dom.broker.impl.DOMNotificationRouter$1.onEvent(DOMNotificationRouter.java:68)[144:org.opendaylight.controller.sal-broker-impl:1.5.1.SNAPSHOT]

(In reply to Sam Hague from comment #5)
> 17-07-10 20:40:26,747 | ERROR | ool-214-thread-5 | NaptEventHandler
> | 342 - org.opendaylight.netvirt.natservice-impl - 0.4.1.SNAPSHOT | NAT
> Service :Exception in NaptEventHandler.handleEvent() payload
> org.opendaylight.netvirt.natservice.internal.NAPTEntryEvent@4a83c660
> java.lang.NullPointerException: the Function passed to Optional.transform()
> must not return null.
> at
> com.google.common.base.Preconditions.checkNotNull(Preconditions.java:226)[38:
> com.google.guava:18.0.0]
> at
> com.google.common.base.Present.transform(Present.java:71)[38:com.google.
> guava:18.0.0]
> at
> org.opendaylight.netvirt.natservice.internal.NatUtil.
> getExtGwMacAddFromRouterId(NatUtil.java:1525)[342:org.opendaylight.netvirt.
> natservice-impl:0.4.1.SNAPSHOT]
> at
> org.opendaylight.netvirt.natservice.internal.NaptEventHandler.
> handleEvent(NaptEventHandler.java:193)[342:org.opendaylight.netvirt.
> natservice-impl:0.4.1.SNAPSHOT]
> at
> org.opendaylight.netvirt.natservice.internal.NaptPacketInHandler$1.
> run(NaptPacketInHandler.java:139)[342:org.opendaylight.netvirt.natservice-
> impl:0.4.1.SNAPSHOT]

The exceptions for NatUtil.getExtGwMacAddFromRouterId is fixed in [1]

[1] https://git.opendaylight.org/gerrit/#/c/60229/

(In reply to Sam Hague from comment #2)
> 2017-07-11 01:32:56,793 | ERROR | eChangeHandler-0 |
> AsyncDataTreeChangeListenerBase | 238 -
> org.opendaylight.genius.mdsalutil-api - 0.3.0.SNAPSHOT | Thread terminated
> due to uncaught exception:
> AsyncDataTreeChangeListenerBase-DataTreeChangeHandler-0
> java.lang.NullPointerException
> at
> org.opendaylight.netvirt.aclservice.AbstractAclServiceImpl.
> processInterfaceUpdate(AbstractAclServiceImpl.java:163)[292:org.opendaylight.
> netvirt.aclservice-impl:0.5.0.SNAPSHOT]
> at
> org.opendaylight.netvirt.aclservice.AbstractAclServiceImpl.
> updateAcl(AbstractAclServiceImpl.java:142)[292:org.opendaylight.netvirt.
> aclservice-impl:0.5.0.SNAPSHOT]
> at
> org.opendaylight.netvirt.aclservice.AclServiceManagerImpl.
> notify(AclServiceManagerImpl.java:57)[292:org.opendaylight.netvirt.
> aclservice-impl:0.5.0.SNAPSHOT]
> at
> org.opendaylight.netvirt.aclservice.listeners.AclInterfaceListener.
> update(AclInterfaceListener.java:104)[292:org.opendaylight.netvirt.
> aclservice-impl:0.5.0.SNAPSHOT]
> at
> org.opendaylight.netvirt.aclservice.listeners.AclInterfaceListener.
> update(AclInterfaceListener.java:33)[292:org.opendaylight.netvirt.aclservice-
> impl:0.5.0.SNAPSHOT]
> at
> org.opendaylight.genius.datastoreutils.
> AsyncDataTreeChangeListenerBase$DataTreeChangeHandler.
> run(AsyncDataTreeChangeListenerBase.java:153)[238:org.opendaylight.genius.
> mdsalutil-api:0.3.0.SNAPSHOT]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:
> 1142)[:1.8.0_131]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:
> 617)[:1.8.0_131]
> at java.lang.Thread.run(Thread.java:748)[:1.8.0_131]

I can see just this one in the latest build
https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-newton-nodl-v2-upstream-stateful-nitrogen/123/odl1_karaf.log.gz

https://git.opendaylight.org/gerrit/#/c/60757/

https://git.opendaylight.org/gerrit/#/c/60301/

The three patches in the comments fixes all the NPE reported here,

[1]https://git.opendaylight.org/gerrit/#/c/60255/
[2]https://git.opendaylight.org/gerrit/#/c/60757/
[3]https://git.opendaylight.org/gerrit/#/c/60301/