OpenDaylight JIRA https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 [OPNFLWPLUG-616] Longevity test rarely goes beyond 6 hours https://jira.opendaylight.org/browse/OPNFLWPLUG-616 OpenFlowPlugin <p>Latest results indicate the switch count is not correct (0!=200).</p> <p><a href="https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-1node-periodic-longevity-lithium-redesign-only-beryllium/buildTimeTrend" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-1node-periodic-longevity-lithium-redesign-only-beryllium/buildTimeTrend</a></p> <p>Operating System: All<br/> Platform: All</p> OPNFLWPLUG-616 Longevity test rarely goes beyond 6 hours Bug Resolved Done Luis Gomez Luis Gomez Mon, 8 Feb 2016 17:26:38 +0000 Mon, 27 Sep 2021 09:01:43 +0000 Mon, 21 Nov 2016 21:20:33 +0000 General 0 3 <p>After analyzing the test results in more detail, the issue seems to be the topology. It does not show any link after a few runs of bringing 200 switches up and down in a linear topology.</p> <p>It seems most of the time longevity test fails is because of this ERROR:</p> <p>2016-09-12 07:00:16,514 | ERROR | pool-32-thread-1 | SalRoleServiceImpl | 183 - org.opendaylight.openflowplugin.impl - 0.3.0.SNAPSHOT | SetRoleService set Role BECOMESLAVE for Node: KeyedInstanceIdentifier</p> {targetType=interface org.opendaylight.yang.gen.v1.urn.opendaylight.inventory.rev130819.nodes.Node, path=[org.opendaylight.yang.gen.v1.urn.opendaylight.inventory.rev130819.Nodes, org.opendaylight.yang.gen.v1.urn.opendaylight.inventory.rev130819.nodes.Node[key=NodeKey [_id=Uri [_value=openflow:33]]]]} <p> fail . Reason java.util.concurrent.CancellationException: Task was cancelled.<br/> 2016-09-12 07:00:17,233 | ERROR | pool-32-thread-1 | SalRoleServiceImpl | 183 - org.opendaylight.openflowplugin.impl - 0.3.0.SNAPSHOT | SetRoleService set Role BECOMESLAVE for Node: KeyedInstanceIdentifier</p> {targetType=interface org.opendaylight.yang.gen.v1.urn.opendaylight.inventory.rev130819.nodes.Node, path=[org.opendaylight.yang.gen.v1.urn.opendaylight.inventory.rev130819.Nodes, org.opendaylight.yang.gen.v1.urn.opendaylight.inventory.rev130819.nodes.Node[key=NodeKey [_id=Uri [_value=openflow:201]]]]} <p> fail . Reason java.util.concurrent.CancellationException: Task was cancelled.<br/> 2016-09-12 07:00:17,235 | ERROR | pool-32-thread-1 | ExecutionList | 41 - com.google.guava - 18.0.0 | RuntimeException while executing runnable com.google.common.util.concurrent.Futures$6@4f23900c with executor INSTANCE<br/> java.lang.NullPointerException<br/> at org.opendaylight.openflowplugin.impl.device.DeviceContextImpl.shutdownConnection(DeviceContextImpl.java:568)<br/> at org.opendaylight.openflowplugin.impl.lifecycle.LifecycleServiceImpl.closeConnection(LifecycleServiceImpl.java:138)<br/> at org.opendaylight.openflowplugin.impl.role.RoleManagerImpl$1.onFailure(RoleManagerImpl.java:86)<br/> at com.google.common.util.concurrent.Futures$6.run(Futures.java:1313)<br/> at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)<br/> at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)<br/> at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)<br/> at com.google.common.util.concurrent.AbstractFuture.cancel(AbstractFuture.java:134)<br/> at org.opendaylight.openflowplugin.impl.role.RoleContextImpl.lambda$sendRoleChangeToDevice$0(RoleContextImpl.java:177)<br/> at io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:581)<span class="error">&#91;115:io.netty.common:4.0.37.Final&#93;</span><br/> at io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:655)<span class="error">&#91;115:io.netty.common:4.0.37.Final&#93;</span><br/> at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:367)<span class="error">&#91;115:io.netty.common:4.0.37.Final&#93;</span><br/> at java.lang.Thread.run(Thread.java:745)<span class="error">&#91;:1.8.0_101&#93;</span></p> <p>This seems to be similar or same ERROR I see in scalability test:</p> <p><a href="https://bugs.opendaylight.org/show_bug.cgi?id=6665" class="external-link" target="_blank" rel="nofollow noopener">https://bugs.opendaylight.org/show_bug.cgi?id=6665</a></p> <p>I am rising priority of this to Critical to be fixed for Boron SR1.</p> <p>Actually, it is exactly same ERROR message as in <a href="https://bugs.opendaylight.org/show_bug.cgi?id=6672" class="external-link" target="_blank" rel="nofollow noopener">https://bugs.opendaylight.org/show_bug.cgi?id=6672</a>. I think I will have patch ready for 6672 today, what will fix cause of that ERROR message, and so it should hopefully fix also this issue.</p> <p>Cool <img class="emoticon" src="https://jira.opendaylight.org/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>This patch should fix this: <a href="https://git.opendaylight.org/gerrit/#/c/45526/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/45526/</a></p> <p>I posted it already in <a href="https://jira.opendaylight.org/browse/OPNFLWPLUG-767" title="Switch connection bounce generates wrong entity owner in cluster env" class="issue-link" data-issue-key="OPNFLWPLUG-767"><del>OPNFLWPLUG-767</del></a> but it fixes it only partially, but it should fix this one (it prevents that NPE exception from occurring).</p> <p>Luis,</p> <p>In the recent links of,<br/> <a href="https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-1node-periodic-longevity-lithium-redesign-only-beryllium/buildTimeTrend" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-1node-periodic-longevity-lithium-redesign-only-beryllium/buildTimeTrend</a></p> <p>the java.lang.NullPointerException and java.util.concurrent.CancellationException does not reproduced in the karaf logs recently.</p> <p>Instead i could see other exceptions as below:</p> <p>2016-09-20 04:15:29,082 | ERROR | lt-dispatcher-16 | RpcBroker | 159 - org.opendaylight.controller.sal-remoterpc-connector - 1.3.4.SNAPSHOT | executeRpc for (urn:opendaylight:packet:service?revision=2013-07-09)transmit-packet failed with root cause: org.opendaylight.controller.md.sal.dom.api.DOMRpcImplementationNotAvailableException: Rpc implementation for {} was removed during processing.. For exception details, enable Debug logging.</p> <p><a href="https://logs.opendaylight.org/releng/jenkins092/openflowplugin-csit-1node-periodic-longevity-lithium-redesign-only-beryllium/398/archives/karaf.log.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/openflowplugin-csit-1node-periodic-longevity-lithium-redesign-only-beryllium/398/archives/karaf.log.gz</a></p> <p>Hope we can open a new bug with mdsal, by closing the current one(5271).</p> <p>In latest Jenkins logs I see VerifyException from Singleton, what is caused by <a href="https://jira.opendaylight.org/browse/MDSAL-197" title="Switch connection bounce results in wrong ownership change in Singleton" class="issue-link" data-issue-key="MDSAL-197"><del>MDSAL-197</del></a>.</p> <p>Read my last reply in <a href="https://jira.opendaylight.org/browse/OPNFLWPLUG-767" title="Switch connection bounce generates wrong entity owner in cluster env" class="issue-link" data-issue-key="OPNFLWPLUG-767"><del>OPNFLWPLUG-767</del></a></p> <p>This is fixed now. Latest run on Boron/Carbon last targeted 23 hours.</p> Blocks MDSAL-197 Development External issue ID 5271 External issue URL ODL SR Target Milestone Rank 0|i03293: