Details
-
Bug
-
Status: Resolved
-
Resolution: Done
-
Carbon
-
None
-
None
-
Operating System: All
Platform: All
-
8160
Description
I ran a test in a loop, that basically creates VMs with FIPs, test connectivity, and deletes them.
After 70 iterations, new VMs connectvity stopped working.
These errors started showing up endlessly:
2017-04-05 03:20:46,877 | INFO | eChangeHandler-0 | LockManager | 351 - org.opendaylight.genius.lockmanager-impl - 0.2.0.SNAPSHOT | Already locked after waiting 1000ms, try 9819
2017-04-05 03:20:47,878 | INFO | eChangeHandler-0 | LockManager | 351 - org.opendaylight.genius.lockmanager-impl - 0.2.0.SNAPSHOT | Already locked after waiting 1000ms, try 9820
2017-04-05 03:20:48,878 | INFO | eChangeHandler-0 | LockManager | 351 - org.opendaylight.genius.lockmanager-impl - 0.2.0.SNAPSHOT | Already locked after waiting 1000ms, try 9821
This is the stack call for this lock:
"AsyncDataTreeChangeListenerBase-DataTreeChangeHandler-0" #4442 daemon prio=5 os_prio=0 tid=0x00007fd1ec08d800 nid=0x5266 waiting on condition [0x00007fd166405000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.opendaylight.lockmanager.LockManager.getLock(LockManager.java:148)
at org.opendaylight.lockmanager.LockManager.lock(LockManager.java:67)
at java.lang.invoke.LambdaForm$DMH/23968864.invokeInterface_LL_L(LambdaForm$DMH)
at java.lang.invoke.LambdaForm$BMH/931340740.reinvoke(LambdaForm$BMH)
at java.lang.invoke.LambdaForm$MH/1202781631.invokeExact_MT(LambdaForm$MH)
at org.opendaylight.yangtools.yang.binding.util.RpcMethodInvokerWithInput.invokeOn(RpcMethodInvokerWithInput.java:31)
at org.opendaylight.yangtools.yang.binding.util.AbstractMappedRpcInvoker.invokeRpc(AbstractMappedRpcInvoker.java:52)
at org.opendaylight.controller.md.sal.binding.impl.BindingDOMRpcImplementationAdapter.invoke(BindingDOMRpcImplementationAdapter.java:83)
at org.opendaylight.controller.md.sal.binding.impl.BindingDOMRpcImplementationAdapter.invokeRpc(BindingDOMRpcImplementationAdapter.java:70)
at org.opendaylight.controller.md.sal.dom.broker.impl.GlobalDOMRpcRoutingTableEntry.invokeRpc(GlobalDOMRpcRoutingTableEntry.java:39)
at org.opendaylight.controller.md.sal.dom.broker.impl.DOMRpcRoutingTable.invokeRpc(DOMRpcRoutingTable.java:177)
at org.opendaylight.controller.md.sal.dom.broker.impl.DOMRpcRouter.invokeRpc(DOMRpcRouter.java:102)
at Proxy83c929da_f599_4e8e_ab68_e2200f2efd8f.invokeRpc(Unknown Source)
at Proxy5c5c056d_30f3_4662_800b_9699ba76b251.invokeRpc(Unknown Source)
at org.opendaylight.controller.md.sal.binding.impl.RpcServiceAdapter.invoke0(RpcServiceAdapter.java:65)
at org.opendaylight.controller.md.sal.binding.impl.RpcServiceAdapter.access$000(RpcServiceAdapter.java:43)
at org.opendaylight.controller.md.sal.binding.impl.RpcServiceAdapter$RpcInvocationStrategy.invoke(RpcServiceAdapter.java:159)
at org.opendaylight.controller.md.sal.binding.impl.RpcServiceAdapter.invoke(RpcServiceAdapter.java:96)
at com.sun.proxy.$Proxy124.lock(Unknown Source)
at org.opendaylight.genius.idmanager.IdUtils.lock(IdUtils.java:230)
at org.opendaylight.genius.idmanager.IdManager.allocateId(IdManager.java:213)
at java.lang.invoke.LambdaForm$DMH/23968864.invokeInterface_LL_L(LambdaForm$DMH)
at java.lang.invoke.LambdaForm$BMH/968378519.reinvoke(LambdaForm$BMH)
at java.lang.invoke.LambdaForm$MH/1202781631.invokeExact_MT(LambdaForm$MH)
at org.opendaylight.yangtools.yang.binding.util.RpcMethodInvokerWithInput.invokeOn(RpcMethodInvokerWithInput.java:31)
at org.opendaylight.yangtools.yang.binding.util.AbstractMappedRpcInvoker.invokeRpc(AbstractMappedRpcInvoker.java:52)
at org.opendaylight.controller.md.sal.binding.impl.BindingDOMRpcImplementationAdapter.invoke(BindingDOMRpcImplementationAdapter.java:83)
at org.opendaylight.controller.md.sal.binding.impl.BindingDOMRpcImplementationAdapter.invokeRpc(BindingDOMRpcImplementationAdapter.java:70)
at org.opendaylight.controller.md.sal.dom.broker.impl.GlobalDOMRpcRoutingTableEntry.invokeRpc(GlobalDOMRpcRoutingTableEntry.java:39)
at org.opendaylight.controller.md.sal.dom.broker.impl.DOMRpcRoutingTable.invokeRpc(DOMRpcRoutingTable.java:177)
at org.opendaylight.controller.md.sal.dom.broker.impl.DOMRpcRouter.invokeRpc(DOMRpcRouter.java:102)
at Proxy83c929da_f599_4e8e_ab68_e2200f2efd8f.invokeRpc(Unknown Source)
at Proxy5c5c056d_30f3_4662_800b_9699ba76b251.invokeRpc(Unknown Source)
at org.opendaylight.controller.md.sal.binding.impl.RpcServiceAdapter.invoke0(RpcServiceAdapter.java:65)
at org.opendaylight.controller.md.sal.binding.impl.RpcServiceAdapter.access$000(RpcServiceAdapter.java:43)
at org.opendaylight.controller.md.sal.binding.impl.RpcServiceAdapter$RpcInvocationStrategy.invoke(RpcServiceAdapter.java:159)
at org.opendaylight.controller.md.sal.binding.impl.RpcServiceAdapter.invoke(RpcServiceAdapter.java:96)
at com.sun.proxy.$Proxy127.allocateId(Unknown Source)
at org.opendaylight.netvirt.natservice.internal.NatUtil.createGroupId(NatUtil.java:773)
at org.opendaylight.netvirt.natservice.internal.RouterDpnChangeListener.removeSNATFromDPN(RouterDpnChangeListener.java:327)
at org.opendaylight.netvirt.natservice.internal.RouterDpnChangeListener.remove(RouterDpnChangeListener.java:193)
at org.opendaylight.netvirt.natservice.internal.RouterDpnChangeListener.remove(RouterDpnChangeListener.java:37)
at org.opendaylight.genius.datastoreutils.AsyncDataTreeChangeListenerBase$DataTreeChangeHandler.run(AsyncDataTreeChangeListenerBase.java:148)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)