[NETCONF-569] NullArgumentException: Context (handler) for /restconf/operational/neutron:neutron/hostconfigs/ is null. Created: 17/Sep/18  Updated: 24/Apr/20

Status: Confirmed
Project: netconf
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Low
Reporter: Sam Hague Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: csit:3node, csit:exception
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File odl1_karaf.log.tar.xz    
Epic Link: Clustering Stability

 Description   

During 3node testing using graceful start and stop the following exception is seen. Graceful start and stop means using the bin/stop and start commands to stop and start ODL rather than using kill -9. This means there is an orderly stop to the bundles where each bundle is destroyed. Some bundles in neutron are destroyed and the exceptions start.

The flow is all three ODLs are up. Then shutdown ODL1 via bin/stop. The exception comes out.

https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/408/shague-haproxy-netvirt-csit-3node-0cmb-1ctl-2cmp-openstack-queens-upstream-stateful-neon/8/odl_1/odl1_karaf.log.gz

2018-09-17T16:51:25,258 | WARN  | qtp200549163-130 | HttpChannel                      | 164 - org.eclipse.jetty.util - 9.3.21.v20170918 | //10.30.170.42:8181/restconf/operational/neutron:neutron/hostconfigs/org.ops4j.lang.NullArgumentException: Context (handler) for /restconf/operational/neutron:neutron/hostconfigs/ is null.
	at org.ops4j.lang.NullArgumentException.validateNotNull(NullArgumentException.java:75) ~[438:org.ops4j.pax.web.pax-web-jetty:6.0.9]
	at org.ops4j.pax.web.service.jetty.internal.JettyServerHandlerCollection.handle(JettyServerHandlerCollection.java:78) [438:org.ops4j.pax.web.pax-web-jetty:6.0.9]
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) [161:org.eclipse.jetty.server:9.3.21.v20170918]
	at org.eclipse.jetty.server.Server.handle(Server.java:534) [161:org.eclipse.jetty.server:9.3.21.v20170918]
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333) [161:org.eclipse.jetty.server:9.3.21.v20170918]
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) [161:org.eclipse.jetty.server:9.3.21.v20170918]
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283) [153:org.eclipse.jetty.io:9.3.21.v20170918]
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108) [153:org.eclipse.jetty.io:9.3.21.v20170918]
	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) [153:org.eclipse.jetty.io:9.3.21.v20170918]
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) [164:org.eclipse.jetty.util:9.3.21.v20170918]
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) [164:org.eclipse.jetty.util:9.3.21.v20170918]
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) [164:org.eclipse.jetty.util:9.3.21.v20170918]
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) [164:org.eclipse.jetty.util:9.3.21.v20170918]
	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) [164:org.eclipse.jetty.util:9.3.21.v20170918]
	at java.lang.Thread.run(Thread.java:748) [?:?]


 Comments   
Comment by Michael Vorburger [ 18/Sep/18 ]

shague while this contains "neutron" it's more of a restconf related issue really, nothing specific to neutron; I'll therefore move it from the neutron to the netconf project.

The problem seems to be more 3rd party (Pax Web) than ODL restconf code related. Investigating, proposing and getting acceptance for and release of a fix in Pax Web JUST for a spurious NPE during shutdown may not be worth the effort here IMHO - this, technically, certainly doesn't really cause any "clustering instability".

Comment by Jakub Morvay [ 01/Oct/18 ]

shague I agree with Michael, this looks more like Pax Web issue and maybe it is not worth the effort here. For now I am going to decrease priority and move this to backlog.

Also maybe we should remove clustering stability epic link.

Comment by Sam Hague [ 08/Oct/18 ]

The problem though is that the bundle is shutting down and still processing incoming requests. so it isn't that there is a NPE - it's that the bundle is still processing requests. It seems the bundle should immediately close listening ports so that no requests can be processed.

This can cause cluster instability in that third-parties which are issuing these RESTCONF requests are getting a negative response back and can mark the ovs nodes as down because the hostconfig is coming back empty.

I would like to keep the clustering stability epic since this looks to be a ODL-wide issue with bundle shutdowns. OFP, restconf, ovsdb all exhibit similar issues where their respective listening ports are still open as the bundles shutdown. Parts of the bundle receive the shutdowns but the listening part is still active.

Generated at Wed Feb 07 20:15:21 UTC 2024 using Jira 8.20.10#820010-sha1:ace47f9899e9ee25d7157d59aa17ab06aee30d3d.