https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/NETVIRT-596 netvirt <p>I ran a test in a loop, that basically creates VMs with FIPs, test connectivity, and deletes them.<br/> After 70 iterations, new VMs connectvity stopped working.</p> <p>These errors started showing up endlessly:<br/> 2017-04-05 03:20:46,877 | INFO | eChangeHandler-0 | LockManager | 351 - org.opendaylight.genius.lockmanager-impl - 0.2.0.SNAPSHOT | Already locked after waiting 1000ms, try 9819<br/> 2017-04-05 03:20:47,878 | INFO | eChangeHandler-0 | LockManager | 351 - org.opendaylight.genius.lockmanager-impl - 0.2.0.SNAPSHOT | Already locked after waiting 1000ms, try 9820<br/> 2017-04-05 03:20:48,878 | INFO | eChangeHandler-0 | LockManager | 351 - org.opendaylight.genius.lockmanager-impl - 0.2.0.SNAPSHOT | Already locked after waiting 1000ms, try 9821</p> <p>This is the stack call for this lock:<br/> "AsyncDataTreeChangeListenerBase-DataTreeChangeHandler-0" #4442 daemon prio=5 os_prio=0 tid=0x00007fd1ec08d800 nid=0x5266 waiting on condition <span class="error">&#91;0x00007fd166405000&#93;</span><br/> java.lang.Thread.State: TIMED_WAITING (sleeping)<br/> at java.lang.Thread.sleep(Native Method)<br/> at org.opendaylight.lockmanager.LockManager.getLock(LockManager.java:148)<br/> at org.opendaylight.lockmanager.LockManager.lock(LockManager.java:67)<br/> at java.lang.invoke.LambdaForm$DMH/23968864.invokeInterface_LL_L(LambdaForm$DMH)<br/> at java.lang.invoke.LambdaForm$BMH/931340740.reinvoke(LambdaForm$BMH)<br/> at java.lang.invoke.LambdaForm$MH/1202781631.invokeExact_MT(LambdaForm$MH)<br/> at org.opendaylight.yangtools.yang.binding.util.RpcMethodInvokerWithInput.invokeOn(RpcMethodInvokerWithInput.java:31)<br/> at org.opendaylight.yangtools.yang.binding.util.AbstractMappedRpcInvoker.invokeRpc(AbstractMappedRpcInvoker.java:52)<br/> at org.opendaylight.controller.md.sal.binding.impl.BindingDOMRpcImplementationAdapter.invoke(BindingDOMRpcImplementationAdapter.java:83)<br/> at org.opendaylight.controller.md.sal.binding.impl.BindingDOMRpcImplementationAdapter.invokeRpc(BindingDOMRpcImplementationAdapter.java:70)<br/> at org.opendaylight.controller.md.sal.dom.broker.impl.GlobalDOMRpcRoutingTableEntry.invokeRpc(GlobalDOMRpcRoutingTableEntry.java:39)<br/> at org.opendaylight.controller.md.sal.dom.broker.impl.DOMRpcRoutingTable.invokeRpc(DOMRpcRoutingTable.java:177)<br/> at org.opendaylight.controller.md.sal.dom.broker.impl.DOMRpcRouter.invokeRpc(DOMRpcRouter.java:102)<br/> at Proxy83c929da_f599_4e8e_ab68_e2200f2efd8f.invokeRpc(Unknown Source)<br/> at Proxy5c5c056d_30f3_4662_800b_9699ba76b251.invokeRpc(Unknown Source)<br/> at org.opendaylight.controller.md.sal.binding.impl.RpcServiceAdapter.invoke0(RpcServiceAdapter.java:65)<br/> at org.opendaylight.controller.md.sal.binding.impl.RpcServiceAdapter.access$000(RpcServiceAdapter.java:43)<br/> at org.opendaylight.controller.md.sal.binding.impl.RpcServiceAdapter$RpcInvocationStrategy.invoke(RpcServiceAdapter.java:159)<br/> at org.opendaylight.controller.md.sal.binding.impl.RpcServiceAdapter.invoke(RpcServiceAdapter.java:96)<br/> at com.sun.proxy.$Proxy124.lock(Unknown Source)<br/> at org.opendaylight.genius.idmanager.IdUtils.lock(IdUtils.java:230)<br/> at org.opendaylight.genius.idmanager.IdManager.allocateId(IdManager.java:213)<br/> at java.lang.invoke.LambdaForm$DMH/23968864.invokeInterface_LL_L(LambdaForm$DMH)<br/> at java.lang.invoke.LambdaForm$BMH/968378519.reinvoke(LambdaForm$BMH)<br/> at java.lang.invoke.LambdaForm$MH/1202781631.invokeExact_MT(LambdaForm$MH)<br/> at org.opendaylight.yangtools.yang.binding.util.RpcMethodInvokerWithInput.invokeOn(RpcMethodInvokerWithInput.java:31)<br/> at org.opendaylight.yangtools.yang.binding.util.AbstractMappedRpcInvoker.invokeRpc(AbstractMappedRpcInvoker.java:52)<br/> at org.opendaylight.controller.md.sal.binding.impl.BindingDOMRpcImplementationAdapter.invoke(BindingDOMRpcImplementationAdapter.java:83)<br/> at org.opendaylight.controller.md.sal.binding.impl.BindingDOMRpcImplementationAdapter.invokeRpc(BindingDOMRpcImplementationAdapter.java:70)<br/> at org.opendaylight.controller.md.sal.dom.broker.impl.GlobalDOMRpcRoutingTableEntry.invokeRpc(GlobalDOMRpcRoutingTableEntry.java:39)<br/> at org.opendaylight.controller.md.sal.dom.broker.impl.DOMRpcRoutingTable.invokeRpc(DOMRpcRoutingTable.java:177)<br/> at org.opendaylight.controller.md.sal.dom.broker.impl.DOMRpcRouter.invokeRpc(DOMRpcRouter.java:102)<br/> at Proxy83c929da_f599_4e8e_ab68_e2200f2efd8f.invokeRpc(Unknown Source)<br/> at Proxy5c5c056d_30f3_4662_800b_9699ba76b251.invokeRpc(Unknown Source)<br/> at org.opendaylight.controller.md.sal.binding.impl.RpcServiceAdapter.invoke0(RpcServiceAdapter.java:65)<br/> at org.opendaylight.controller.md.sal.binding.impl.RpcServiceAdapter.access$000(RpcServiceAdapter.java:43)<br/> at org.opendaylight.controller.md.sal.binding.impl.RpcServiceAdapter$RpcInvocationStrategy.invoke(RpcServiceAdapter.java:159)<br/> at org.opendaylight.controller.md.sal.binding.impl.RpcServiceAdapter.invoke(RpcServiceAdapter.java:96)<br/> at com.sun.proxy.$Proxy127.allocateId(Unknown Source)<br/> at org.opendaylight.netvirt.natservice.internal.NatUtil.createGroupId(NatUtil.java:773)<br/> at org.opendaylight.netvirt.natservice.internal.RouterDpnChangeListener.removeSNATFromDPN(RouterDpnChangeListener.java:327)<br/> at org.opendaylight.netvirt.natservice.internal.RouterDpnChangeListener.remove(RouterDpnChangeListener.java:193)<br/> at org.opendaylight.netvirt.natservice.internal.RouterDpnChangeListener.remove(RouterDpnChangeListener.java:37)<br/> at org.opendaylight.genius.datastoreutils.AsyncDataTreeChangeListenerBase$DataTreeChangeHandler.run(AsyncDataTreeChangeListenerBase.java:148)<br/> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)<br/> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)<br/> at java.lang.Thread.run(Thread.java:745)</p> <p>Operating System: All<br/> Platform: All</p> NETVIRT-596

getLock() invoked from createGroupId is caught up in an endless loop - can't connect to new L3 VMs

Bug Resolved Done Unassigned Koby Aizer Wed, 5 Apr 2017 14:34:54 +0000 Thu, 19 Oct 2017 21:28:04 +0000 Tue, 25 Apr 2017 13:32:05 +0000 Carbon General 0 4 <p>Attachment getLock.zip has been added with description: logs</p> <p>Kency would raise a review shortly with a potential fix (done through code study) and more info logs to figure out exact root cause.</p> <p>Thanks!</p> <p>Another thought here - even if we fix the root cause, I think we must remove this busy loop. This totally breaks the system when occurring</p> <p>Looks like this issue occurred when an idKey is not unlocked accidentally in IdManager which leads to LockManager#lock goes into busy spin. But this is a corner case due to a bug.</p> <p>We could avoid it by thread wait/notify by Async Clustered DTCN on lock (since lock is stored in MD-SAL), upon remove DCN, waiting threads can be notified for the particular lock. Not sure its worth to do it now and should be done as an enhancement since it needs more testing effort.</p> <p>Raised the review for this:<br/> <a href="https://git.opendaylight.org/gerrit/#/c/54414/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/54414/</a></p> <p>Thanks guys for handling this.. Could've taken me a long time to find it myself <img class="emoticon" src="https://jira.opendaylight.org/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>Should we open another ticket on removing the busy loop future fix?</p> Development External issue ID 8160 External issue URL Rank 0|i01srz: Status Whiteboard csit:sporadic_failures