https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/NETVIRT-1132 netvirt <p><a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-openstack-queens-upstream-stateful-oxygen/191/odl_1/odl1_karaf.log.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-openstack-queens-upstream-stateful-oxygen/191/odl_1/odl1_karaf.log.gz</a></p> <p>The exception happens in the add and update paths. Add path is here and later is the update path:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>2018-03-06T05:23:47,604 | INFO | org.opendaylight.yang.gen.v1.urn.opendaylight.genius.itm.op.rev160406.tunnels_state.StateTunnelList_AsyncDataTreeChangeListenerBase-DataTreeChangeHandler-0 | TunnelInterfaceStateListener | 367 - org.opendaylight.netvirt.vpnmanager-impl - 0.6.0.SNAPSHOT | handleTunnelEventForDPN: Handle tunnel event for srcDpn 110780754426209 SrcTepIp 10.30.170.87 DestTepIp 10.30.170.102 2018-03-06T05:23:47,604 | INFO | org.opendaylight.yang.gen.v1.urn.opendaylight.genius.itm.op.rev160406.tunnels_state.StateTunnelList_AsyncDataTreeChangeListenerBase-DataTreeChangeHandler-0 | TunnelInterfaceStateListener | 367 - org.opendaylight.netvirt.vpnmanager-impl - 0.6.0.SNAPSHOT | handleTunnelEventForDPN: Tunnel ADD event received for Dpn 110780754426209 VTEP Ip 10.30.170.87 destTepIp 2018-03-06T05:23:47,604 | INFO | ForkJoinPool-1-worker-1 | ElanTunnelInterfaceStateListener | 348 - org.opendaylight.netvirt.elanmanager-impl - 0.6.0.SNAPSHOT | Handling tunnel state event for srcDpId 110780754426209 and dstDpId 106567621600656 2018-03-06T05:23:47,605 | ERROR | org.opendaylight.yang.gen.v1.urn.opendaylight.genius.itm.op.rev160406.tunnels_state.StateTunnelList_AsyncDataTreeChangeListenerBase-DataTreeChangeHandler-0 | TunnelInterfaceStateListener | 367 - org.opendaylight.netvirt.vpnmanager-impl - 0.6.0.SNAPSHOT | handleTunnelEventForDpn: Unable to handle the tunnel event for srcDpnId 110780754426209 srcTepIp 10.30.170.87 remoteDpnId 106567621600656 destTepIp 10.30.170.102 java.lang.NullPointerException: null at org.opendaylight.netvirt.vpnmanager.TunnelInterfaceStateListener.handleTunnelEventForDPN(TunnelInterfaceStateListener.java:340) [367:org.opendaylight.netvirt.vpnmanager-impl:0.6.0.SNAPSHOT] at org.opendaylight.netvirt.vpnmanager.TunnelInterfaceStateListener.add(TunnelInterfaceStateListener.java:217) [367:org.opendaylight.netvirt.vpnmanager-impl:0.6.0.SNAPSHOT] at org.opendaylight.netvirt.vpnmanager.TunnelInterfaceStateListener.add(TunnelInterfaceStateListener.java:76) [367:org.opendaylight.netvirt.vpnmanager-impl:0.6.0.SNAPSHOT] at org.opendaylight.genius.datastoreutils.AsyncDataTreeChangeListenerBase$DataTreeChangeHandler.run(AsyncDataTreeChangeListenerBase.java:167) [260:org.opendaylight.genius.mdsalutil-api:0.4.0.SNAPSHOT] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:?] at java.lang.Thread.run(Thread.java:748) [?:?]</pre> </div></div> <p>This also happens in the update path:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>2018-03-06T05:23:55,042 | INFO | org.opendaylight.yang.gen.v1.urn.opendaylight.genius.itm.op.rev160406.tunnels_state.StateTunnelList_AsyncDataTreeChangeListenerBase-DataTreeChangeHandler-0 | TunnelInterfaceStateListener | 367 - org.opendaylight.netvirt.vpnmanager-impl - 0.6.0.SNAPSHOT | handleTunnelEventForDPN: Tunnel ADD event received for Dpn 110780754426209 VTEP Ip 10.30.170.87 destTepIp 2018-03-06T05:23:55,043 | ERROR | org.opendaylight.yang.gen.v1.urn.opendaylight.genius.itm.op.rev160406.tunnels_state.StateTunnelList_AsyncDataTreeChangeListenerBase-DataTreeChangeHandler-0 | TunnelInterfaceStateListener | 367 - org.opendaylight.netvirt.vpnmanager-impl - 0.6.0.SNAPSHOT | handleTunnelEventForDpn: Unable to handle the tunnel event for srcDpnId 110780754426209 srcTepIp 10.30.170.87 remoteDpnId 106567621600656 destTepIp 10.30.170.102 java.lang.NullPointerException: null at org.opendaylight.netvirt.vpnmanager.TunnelInterfaceStateListener.handleTunnelEventForDPN(TunnelInterfaceStateListener.java:340) [367:org.opendaylight.netvirt.vpnmanager-impl:0.6.0.SNAPSHOT] at org.opendaylight.netvirt.vpnmanager.TunnelInterfaceStateListener.update(TunnelInterfaceStateListener.java:169) [367:org.opendaylight.netvirt.vpnmanager-impl:0.6.0.SNAPSHOT] at org.opendaylight.netvirt.vpnmanager.TunnelInterfaceStateListener.update(TunnelInterfaceStateListener.java:76) [367:org.opendaylight.netvirt.vpnmanager-impl:0.6.0.SNAPSHOT] at org.opendaylight.genius.datastoreutils.AsyncDataTreeChangeListenerBase$DataTreeChangeHandler.run(AsyncDataTreeChangeListenerBase.java:172) [260:org.opendaylight.genius.mdsalutil-api:0.4.0.SNAPSHOT] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:?] at java.lang.Thread.run(Thread.java:748) [?:?]</pre> </div></div> NETVIRT-1132

NPE: StateTunnelList_ADTCL handleTunnelEventForDpn: Unable to handle the tunnel event for srcDpnId 110780754426209 srcTepIp 10.30.170.87 remoteDpnId 106567621600656 destTepIp 10.30.170.102 handleTunnelEventForDPN(TunnelInterfaceStateListener.java:340)

Bug Medium Resolved Done Victor Pickard Sam Hague csit:exception patch_merged Tue, 6 Mar 2018 14:30:02 +0000 Tue, 13 Mar 2018 19:42:03 +0000 Tue, 13 Mar 2018 19:42:03 +0000 Oxygen Oxygen General 0 2 <p>OK, here are my observations from looking into this exception. Sam, thanks for forwarding the email thread about neutron creating initial networks being enabled. That was a good clue to help narrow down this issue.<br/>  <br/> Here is the sandbox job <span class="error">&#91;1&#93;</span> I ran with extra debug to get the interface name that was not found in config d/s that was causing the exception.<br/>  <br/>  <br/> *+Why is the interface ad7be255-1fc2-4bab-93e1-9641504c27fb not found in config d/s? <br/> +* <br/> TunnelInterfaceStateListener::handleTunnelEventForDPN makes a call to genius InterfaceManager::getDpnInterfaceList(), which returns interfaces from OPERATIONAL d/s, NOT config d/s. </p> <p>It appears that this port was never added to config d/s..... perhaps because neutron create initial networks was enabled?? </p> <p>So, we get a list of interfaces from genius from oper d/s, and then look for the interface (netvirt) in config d/s, and it is not there. This doesn't seem correct, should genius be returning interfaces from config d/s? I suppose a more general question, how can a port end up in oper d/s and not be in config d/s? Is it because neutron created the port before ODL was up, and did not receive the neutron port create event? Probably so..... <br/>  <br/>  <br/> <b><ins>Debugging details</ins></b><br/>  </p> <p><b><ins>Interface that is not found in config d/s ad7be255-1fc2-4bab-93e1-9641504c27fb</ins></b> <br/> <font color="#000000"></font>2018-03-07T20:34:26,183 | INFO  | org.opendaylight.yang.gen.v1.urn.opendaylight.genius.itm.op.rev160406.tunnels_state.StateTunnelList_AsyncDataTreeChangeListenerBase-Data<br/> TreeChangeHandler-0 | TunnelInterfaceStateListener     | 364 - org.opendaylight.netvirt.vpnmanager-impl - 0.7.0.SNAPSHOT | handleTunnelEventForDPN: Processing Interface n<br/> ame ad7be255-1fc2-4bab-93e1-9641504c27fb, Interface Interfaces [_interfaceName=ad7be255-1fc2-4bab-93e1-9641504c27fb, _interfaceType=class org.opendaylight.yang.gen.v1.urn<br/> .ietf.params.xml.ns.yang.iana._if.type.rev140508.L2vlan, augmentation=[]]</p> <p>2018-03-07T20:34:26,183 | ERROR | org.opendaylight.yang.gen.v1.urn.opendaylight.genius.itm.op.rev160406.tunnels_state.StateTunnelList_AsyncDataTreeChangeListenerBase-Data<br/> TreeChangeHandler-0 | TunnelInterfaceStateListener     | 364 - org.opendaylight.netvirt.vpnmanager-impl - 0.7.0.SNAPSHOT | handleTunnelEventForDpn: Unable to handle the t<br/> unnel event for srcDpnId 277980448198840 srcTepIp 10.30.170.83 remoteDpnId 154421049938961 destTepIp 10.30.170.96 <br/> java.lang.NullPointerException: null <br/>        at org.opendaylight.netvirt.vpnmanager.TunnelInterfaceStateListener.handleTunnelEventForDPN(TunnelInterfaceStateListener.java:343) [364:org.opendaylight.netvirt.v<br/> pnmanager-impl:0.7.0.SNAPSHOT] <br/>        at org.opendaylight.netvirt.vpnmanager.TunnelInterfaceStateListener.add(TunnelInterfaceStateListener.java:219) [364:org.opendaylight.netvirt.vpnmanager-impl:0.7.0<br/> .SNAPSHOT] <br/>        at org.opendaylight.netvirt.vpnmanager.TunnelInterfaceStateListener.add(TunnelInterfaceStateListener.java:76) [364:org.opendaylight.netvirt.vpnmanager-impl:0.7.0.<br/> SNAPSHOT] <br/>        at org.opendaylight.genius.datastoreutils.AsyncDataTreeChangeListenerBase$DataTreeChangeHandler.run(AsyncDataTreeChangeListenerBase.java:167) [258:org.opendayligh<br/> t.genius.mdsalutil-api:0.5.0.SNAPSHOT] <br/>        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) <span class="error">&#91;?:?&#93;</span> <br/>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) <span class="error">&#91;?:?&#93;</span> <br/>        at java.lang.Thread.run(Thread.java:748) <span class="error">&#91;?:?&#93;</span> <br/>  </p> <p><b><ins>1st robot test... exception comes out even before 1st test starts... is this related to neutron creating initial networks?</ins></b> <br/> 2018-03-07T20:40:17,258 | INFO  | pipe-log:log "ROBOT MESSAGE: Starting suite /w/workspace/netvirt-csit-1node-openstack-queens-vic-upstream-stateful-fluorine/test/csit/su<br/> ites/openstack/connectivity/l2.robot" | core                             | 123 - org.apache.karaf.log.core - 4.1.3 | ROBOT MESSAGE: Starting suite /w/workspace/netvirt-cs<br/> it-1node-openstack-queens-vic-upstream-stateful-fluorine/test/csit/suites/openstack/connectivity/l2.robot <br/> 2018-03-07T20:40:27,363 | INFO  | pipe-log:log "ROBOT MESSAGE: Starting suite /w/workspace/netvirt-csit-1node-openstack-queens-vic-upstream-stateful-fluorine/test/csit/su<br/> ites/openstack/connectivity/l2.robot" | core                             | 123 - org.apache.karaf.log.core - 4.1.3 | ROBOT MESSAGE: Starting test Create VLAN Network net_<br/> 1 </p> <p><b><ins>When is the interface ad7be255-1fc2-4bab-93e1-9641504c27fb created?</ins></b> </p> <p>2018-03-07T20:34:05,041 | INFO  | ForkJoinPool-1-worker-6 | NeutronPortChangeListener        | 356 - org.opendaylight.netvirt.neutronvpn-impl - 0.7.0.SNAPSHOT | Of-port-interface creation for port ad7be255-1fc2-4bab-93e1-9641504c27fb </p> <p>Port created a good 6+ minutes before any robot csit tests start... </p> <p>2018-03-07T20:34:06,719 | INFO  | org.opendaylight.yang.gen.v1.urn.opendaylight.neutron.ports.rev150712.ports.attributes.ports.Port_AsyncClusteredDataTreeChangeListenerBa<br/> se-DataTreeChangeHandler-0 | NeutronPortChangeListener        | 350 - org.opendaylight.netvirt.ipv6service-impl - 0.7.0.SNAPSHOT | IPv6: addInterfaceInfo is invoked for ahost interface Port{getAllowedAddressPairs=[], getDeviceId=dhcpe5510a36-5d54-52f9-9732-333c3a36fa80-0e1251ed-f2a1-42b9-940e-1c0f5341e75d, getDeviceOwner=network:dhcp, getExtraDhcpOpts=[], getFixedIps=[FixedIps{getIpAddress=IpAddress [_ipv4Address=Ipv4Address <span class="error">&#91;_value=10.0.0.2&#93;</span>], getSubnetId=Uuid [_value=b8b43068-d1cd-49b6-88f0-9c6db910d37<br/> 5], augmentations={}}, FixedIps{getIpAddress=IpAddress [_ipv6Address=Ipv6Address <span class="error">&#91;_value=fd12:5ac9:f253:0:f816:3eff:fe6e:aa7d&#93;</span>], getSubnetId=Uuid <span class="error">&#91;_value=c2b3a176-3742-45a5-ba3d-7302d0ddf2b7&#93;</span>, augmentations={}}], getMacAddress=MacAddress <span class="error">&#91;_value=fa:16:3e:6e:aa:7d&#93;</span>, getName=, getNetworkId=Uuid <span class="error">&#91;_value=0e1251ed-f2a1-42b9-940e-1c0f5341e75d&#93;</span>,<br/> getRevisionNumber=7, getSecurityGroups=[], getTenantId=Uuid <span class="error">&#91;_value=b76b365c-9351-482f-9f5c-beb8570e29a4&#93;</span>, getUuid=Uuid <span class="error">&#91;_value=ad7be255-1fc2-4bab-93e1-9641504c27fb&#93;</span>, isAdminStateUp=true, augmentations={interfaceorg.opendaylight.yang.gen.v1.urn.opendaylight.neutron.portsecurity.rev150712.PortSecurityExtension=PortSecurityExtension</p> {isPor tSecurityEnabled=false} <p>,interfaceorg.opendaylight.yang.gen.v1.urn.opendaylight.neutron.binding.rev150712.PortBindingExtension=PortBindingExtension{getHostId=<a href="http://sandbox-31488-4-1-devstack-pike-0.vexxhost.net/" class="external-link" target="_blank" rel="nofollow noopener">sandbox-31488-4-1-devstack-pike-0.vexxhost.net</a>, getProfile={}, getVifDetails=<span class="error">&#91;VifDetails\{getDetailsKey=support_vhost_user, getValue=false, augmentations={}}, VifDetails\{getDetailsKey=uuid, getValue=46f51f05-7822-4af1-a99d-03e0f03cdd59, augmentations={}}, VifDetails\{getDetailsKey=has_datapath_type_netdev, getValue=false, augmentations={}}&#93;</span>, getVifTyp<br/> e=ovs, getVnicType=normal}}}, fixedIp: FixedIps{getIpAddress=IpAddress [_ipv6Address=Ipv6Address <span class="error">&#91;_value=fd12:5ac9:f253:0:f816:3eff:fe6e:aa7d&#93;</span>], getSubnetId=Uuid <span class="error">&#91;_value=c2b3a176-3742-45a5-ba3d-7302d0ddf2b7&#93;</span>, augmentations={}}</p> <p><b><ins>Robot L2 suite runs. Openstack Suite setup cleans up, and removes the port</ins></b> </p> <p><a href="https://logs.opendaylight.org/sandbox/vex-yul-odl-jenkins-2/netvirt-csit-1node-openstack-queens-vic-upstream-stateful-fluorine/4/robot-plugin/log_01_l2.html.gz#s1-k2-k1-k" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/sandbox/vex-yul-odl-jenkins-2/netvirt-csit-1node-openstack-queens-vic-upstream-stateful-fluorine/4/robot-plugin/log_01_l2.html.gz#s1-k2-k1-k</a><br/> 7-k4 </p> <p>KEYWORD BuiltIn . Log ${list} <br/> Documentation: <br/> Logs the given message with the given level. </p> <p>Start / End / Elapsed:  20180307 15:47:06.683 / 20180307 15:47:06.684 / 00:00:00.001 <br/> 15:47:06.684    INFO    [</p> {u'Status': u'ACTIVE', u'Fixed IP Addresses': u"ip_address='10.0.0.2', subnet_id='b8b43068-d1cd-49b6-88f0-9c6db910d375'\nip_address='fd12:5ac9:f2 53:0:f816:3eff:fe6e:aa7d', subnet_id='c2b3a176-3742-45a5-ba3d-7302d0ddf2b7'", u'ID': u'ad7be255-1fc2-4bab-93e1-9641504c27fb', u'MAC Address': u'fa:16:3e:6e:aa:7d', u'Name ': u''} <p>] </p> <p>Start / End / Elapsed:  20180307 15:47:06.708 / 20180307 15:47:08.388 / 00:00:01.680 <br/> 15:47:06.710    INFO    Running command 'openstack port delete ad7be255-1fc2-4bab-93e1-9641504c27fb 2&gt;&amp;1'.<br/>  <br/>  <br/> <span class="error">&#91;1&#93;</span> <a href="https://logs.opendaylight.org/sandbox/vex-yul-odl-jenkins-2/netvirt-csit-1node-openstack-queens-vic-upstream-stateful-fluorine/4/odl_1/odl1_karaf.log.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/sandbox/vex-yul-odl-jenkins-2/netvirt-csit-1node-openstack-queens-vic-upstream-stateful-fluorine/4/odl_1/odl1_karaf.log.gz</a><br/>  </p> <p>OK, the previous analysis was incorrect. Here is the latest update, thanks for helping with this Michael/Stephen/Sam!</p> <p> </p> <p> <br/> Victor,<br/>  <br/> as discussed today on #opendaylight-netvirt, and with full due credit to Sam and Stephen, "here's what happened", as Mr. Monk would say:<br/>  <br/>  <br/> The NPE at line 343 of netvirt.vpnmanager.TunnelInterfaceStateListener (that's <b>WITH</b> your <a href="https://git.opendaylight.org/gerrit/#/c/69222/2/vpnmanager/impl/src/main/java/org/opendaylight/netvirt/vpnmanager/TunnelInterfaceStateListener.java" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/69222/2/vpnmanager/impl/src/main/java/org/opendaylight/netvirt/vpnmanager/TunnelInterfaceStateListener.java</a>; on master that is 341, this caused me a bit of confusion initially) in this line:<br/>  <br/>                 if (vpnInterface != null &amp;&amp; !vpnInterface.isScheduledForRemove()) {<br/>  <br/> is caused by Boolean isScheduledForRemove() - note the object Boolean instead of the primitive boolean. It's because of what's called "auto boxing" in Java, and not that obvious to spot... basically a boolean can be (only) true or false but a Boolean can that OR null, and in an if (theBoolean) you can get an NPE wherever it has to get "coerced" to a boolean (like in an if, but e.g. !vpnInterface.isScheduledForRemove() etc. could cause it as well).<br/>  <br/> A trivial fix could be to just change that to this, as suggested by Stephen:<br/>                  if (vpnInterface != null &amp;&amp; !Boolean.TRUE.equals(vpnInterface.isScheduledForRemove())) {<br/>  <br/> which, just for illustration basicaly just does this:<br/>  <br/>                 if (vpnInterface != null &amp;&amp; vpnInterface.isScheduledForRemove() != null &amp;&amp; !vpnInterface.isScheduledForRemove()) {<br/>  <br/> BUT I would question whether it's expected to get null in there - whever that vpnInterface was created, is there a setScheduledForRemove() call missing on a VpnInterfaceBuilder somewhere? in the YANG model for this, should the scheduledForRemove perhaps default to false? Or we would you’d need to add “mandatory” to the YANG model?<br/>  <br/> Another angle would be to do something on the code quality metrics side of things to see if we could fail on unchecked possible Boolean usage as boolean. I'm not sure if such a check already exists, but will have a look.<br/>  </p> <p>Michael/Stephen/Sam,<br/> Thanks for taking a look at this NPE and providing the details, especially the explanation about "auto boxing", I clearly didn't see it in the code, your analysis is much appreciated.</p> <p>As Sam noted in #opendaylight-netvirt, this vpnInterface.isScheduledForRemove() came in a recent patch, <a href="https://git.opendaylight.org/gerrit/#/c/68151/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/68151/</a>.</p> <p>I have pushed a patch <span class="error">&#91;0&#93;</span> that makes the call to setScheduledForRemove() on the VpnInterface when it is created for all references, and also, added the extra null check as suggested by Stephen in TunnelInterfaceStateListener.java.</p> <p>I'll add Kiran and Phillipe to the review.</p> <p><span class="error">&#91;0&#93;</span> <a href="https://git.opendaylight.org/gerrit/69398" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/69398</a></p> <p><a href="https://git.opendaylight.org/gerrit/69398" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/69398</a></p> Development Rank 0|i03b87: