https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/CONTROLLER-1224 controller <p>Jozez Behran created a test job that repeatedly performs a small load test of Helium SR3 during attempt to push 5k paths to ODL through BGP. </p> <p>It exposes a bug which causes the odl-restconf-noauth to hang on install occasionally. </p> <p>He has captured a YourKit performance snapshot in the time when the test failed due to the hang so it is now possible to see what exactly is blocked and where.</p> <p>I'll add as attachment.</p> <p>Operating System: All<br/> Platform: All</p> CONTROLLER-1224

An attempt to push 5k paths to ODL SR3 through BGP exposed a bug which causes the odl-restconf-noauth to hang on install approx 3% of the time

Bug Resolved Won't Do Jozef Behran RichardHill Tue, 24 Mar 2015 16:40:27 +0000 Thu, 23 Jul 2015 09:49:33 +0000 Thu, 23 Jul 2015 09:49:33 +0000 Helium restconf 0 3 <p><a href="https://drive.google.com/file/d/0ByXiyf4iY7RYbXZWdlVYdHV6clE/view?usp=sharing" class="external-link" target="_blank" rel="nofollow noopener">https://drive.google.com/file/d/0ByXiyf4iY7RYbXZWdlVYdHV6clE/view?usp=sharing</a></p> <p>Needs yourkit profiler to view</p> <p><a href="https://drive.google.com/file/d/0ByXiyf4iY7RYTjVQMFpNV2ExZmc/view?usp=sharing" class="external-link" target="_blank" rel="nofollow noopener">https://drive.google.com/file/d/0ByXiyf4iY7RYTjVQMFpNV2ExZmc/view?usp=sharing</a></p> <p><a href="https://drive.google.com/file/d/0ByXiyf4iY7RYX2Y0c3hVdTF6NFU/view?usp=sharing" class="external-link" target="_blank" rel="nofollow noopener">https://drive.google.com/file/d/0ByXiyf4iY7RYX2Y0c3hVdTF6NFU/view?usp=sharing</a></p> <p>This test executes a performance test of Helium SR3 on the following scenario:</p> <p> Unclustered configuration<br/> Topology and RIB are both updated<br/> IMDS used<br/> 5 k paths</p> <p>Investigation of profiling snapshot revealed the suspicious method is <br/> in file "OsgiRegistry.java" OsgiRegistry.setOSGi.ServiceFinderIteratorProvider</p> <p>My suspection is that odl-restconf-noauth is doing something wrong with the OSGi registry (or with the Karaf's InstallFeature functionality) and when the constellation of the bytes in the process happens to be just right, it hangs the whole thing.</p> <p>I found the following:</p> <ul class="alternate" type="square"> <li>There is a thread with name pool-X-thread-Y (X and Y are some numbers that are differing from run to run) which is "runnable" and its stack trace looks like this:</li> </ul> <p> com.sun.jersey.core.osgi.OsgRegistry.setOSGiServiceFinderIteratorProvider()<br/> at OsgiRegistry.java:422<br/> org.apache.karaf.features.internal.FeaturesServiceImpl.installFeature(String, String, EnumSet)<br/> at FeaturesServiceImpl.java:362<br/> Proxy&lt;some_horrible_number_in_GUID_format&gt;.installFeature(String, String, EnumSet)<br/> at ??? (maybe somehow autogenerated ???)<br/> java.lang.Thread.run()</p> <p> To me this thread seems to be the one trying to install odl-restconf-noauth</p> <ul class="alternate" type="square"> <li>There is another thread with name pool-X-thread-Y (X and Y are some numbers that are differing from run to run and differing from the first thread) which is blocked here:</li> </ul> <p> com.sun.jersey.core.osgi.OsgiRegistry.getInstance()<br/> at OsgiRegistry.java:113<br/> org.eclipse.jetty.server.handler.ContextHandler.doStart()<br/> at ContextHandler.java:717<br/> org.ops4j.pax.web.service.jetty.internal.HttpServiceContext.doStart()<br/> at HttpServiceContext.java:222<br/> org.eclipse.jetty.util.component.AbstractLifeCycle.start()<br/> at AbstractLifeCycle.java:64<br/> org.ops4j.pax.web.service.jetty.internal.JettyServerImpl$1.start()<br/> at jettyServerImpl.java:197<br/> org.ops4j.pax.web.service.internal.HttpServiceStarted.end(HttpContext)<br/> at HttpServiceStarted.java:1032<br/> org.ops4j.pax.web.service.internal.HttpServiceProxy.end(HttpContext)<br/> at HttpServiceProxy.java:422<br/> org.ops4j.pax.web.extender.war.internal.RegisterWebAppVisitorWC.end()<br/> at RegisterWebAppVisitorWC.java:341<br/> org.ops4j.pax.web.extender.war.internal.model.WebApp.accept(WebAppVisitor)<br/> at WebApp.java:678<br/> org.ops4j.pax.web.extender.war.internal.WebAppPublisher$WebAppDependencyListener.register(WebAppDependencyHolder, HttpService)<br/> at WebAppPublisher.java:237<br/> org.ops4j.pax.web.extender.war.internal.WebAppPublisher$WebAppDependencyListener.addingService(ServiceReference)<br/> at WebAppPublisher.java:182<br/> org.ops4j.pax.web.extender.war.internal.WebAppPublisher$WebAppDependencyListener.addingService(ServiceReference)<br/> at WebAppPublisher.java:135<br/> org.osgi.util.tracker.ServiceTracker.open()<br/> at ServiceTracker.java:261<br/> org.ops4j.pax.web.extender.war.internal.WebAppPublisher.publish(WebApp)<br/> at WebAppPublisher.java:101<br/> org.ops4j.pax.web.extender.war.internal.WebObserver.deploy(WebApp)<br/> at WebObserver.java:213<br/> org.ops4j.pax.web.extender.war.internal.WebObserver$1.doStart()<br/> at WebObserver.java:175<br/> org.ops4j.pax.web.extender.war.internal.extender.SimpleExtension.start()<br/> at SimpleExtension.java:58<br/> org.ops4j.pax.web.extender.war.internal.extender.AbstractExtender$1.run()<br/> at AbstractExtender.java:266<br/> java.lang.Thread.run()</p> <p>I am not sure about the purpose of this thread or why it is still hanging around here but in every failure I see it hung at exactly this one spot. I was not able to track this thread's stacktrace down in a snapshot of a build where the test passed though.</p> <p>Hi all.<br/> I'd like to ask for all steps for a possibility to reproduction the described behavior. Please provide some test inputs too (e.g. robot test path or file or something).<br/> I'd like to ask for a whole log file, because it could by a problem with loading required dependencies (follow J.Behran description) but in some step before. Did you try to reload odl-restconf-noauth in the described unexpected state?</p> <p>Hi Vaclav </p> <p>To clarify you want the test rerun with full logging and the log provided.</p> <p>Testing for this bug requires running the test job for several days. Additionally, the full logging is most likely to cause this type of heisenbug to disappear. As Helium SR4 is the last release and its deadline is today, there is no time to even determine whether the bug is still there or not, let alone fixing it.</p> Development External issue ID 2891 External issue URL ODL SR Target Milestone Rank 0|i02p93: