https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/NETVIRT-727 netvirt <p>Seen in <a href="https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-newton-nodl-v2-upstream-stateful-carbon/673/odl1_karaf.log.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-newton-nodl-v2-upstream-stateful-carbon/673/odl1_karaf.log.gz</a></p> <p>2017-06-13 04:28:28,597 | INFO | eChangeHandler-0 | LockListener | 317 - org.opendaylight.genius.lockmanager-impl - 0.2.1.SNAPSHOT | Received add for lock acl.flow.priorities.pool.123829938003760.243.PERMITTCP_DESTINATION_1_0Ingress33216dc65c-dab2-44e4-a649-d5b3d094003c : Lock{getLockName=acl.flow.priorities.pool.123829938003760.243.PERMITTCP_DESTINATION_1_0Ingress33216dc65c-dab2-44e4-a649-d5b3d094003c, getLockOwner=AsyncDataTreeChangeListenerBase-DataTreeChangeHandler-0, augmentations={}}<br/> 2017-06-13 04:28:28,598 | ERROR | eChangeHandler-0 | IdManager | 318 - org.opendaylight.genius.idmanager-impl - 0.2.1.SNAPSHOT | releaseId failed: ReleaseIdInput [_idKey=TCP_DESTINATION_1_0Ingress33216dc65c-dab2-44e4-a649-d5b3d094003c, _poolName=acl.flow.priorities.pool.123829938003760.243.PERMIT, augmentation=[]]<br/> org.opendaylight.genius.idmanager.IdManagerException: Id Entries does not exist<br/> at org.opendaylight.genius.idmanager.IdManager.releaseIdFromLocalPool(IdManager.java:628)<span class="error">&#91;318:org.opendaylight.genius.idmanager-impl:0.2.1.SNAPSHOT&#93;</span><br/> at org.opendaylight.genius.idmanager.IdManager.releaseId(IdManager.java:294)<span class="error">&#91;318:org.opendaylight.genius.idmanager-impl:0.2.1.SNAPSHOT&#93;</span><br/> at org.opendaylight.yangtools.yang.binding.util.RpcMethodInvokerWithInput.invokeOn(RpcMethodInvokerWithInput.java:31)<span class="error">&#91;47:org.opendaylight.mdsal.yang-binding:0.10.1.SNAPSHOT&#93;</span><br/> at org.opendaylight.yangtools.yang.binding.util.AbstractMappedRpcInvoker.invokeRpc(AbstractMappedRpcInvoker.java:52)<span class="error">&#91;47:org.opendaylight.mdsal.yang-binding:0.10.1.SNAPSHOT&#93;</span><br/> at org.opendaylight.controller.md.sal.binding.impl.BindingDOMRpcImplementationAdapter.invoke(BindingDOMRpcImplementationAdapter.java:83)<span class="error">&#91;146:org.opendaylight.controller.sal-binding-broker-impl:1.5.1.SNAPSHOT&#93;</span><br/> at org.opendaylight.controller.md.sal.binding.impl.BindingDOMRpcImplementationAdapter.invokeRpc(BindingDOMRpcImplementationAdapter.java:70)<span class="error">&#91;146:org.opendaylight.controller.sal-binding-broker-impl:1.5.1.SNAPSHOT&#93;</span><br/> at org.opendaylight.controller.md.sal.dom.broker.impl.GlobalDOMRpcRoutingTableEntry.invokeRpc(GlobalDOMRpcRoutingTableEntry.java:39)<span class="error">&#91;144:org.opendaylight.controller.sal-broker-impl:1.5.1.SNAPSHOT&#93;</span><br/> at org.opendaylight.controller.md.sal.dom.broker.impl.DOMRpcRoutingTable.invokeRpc(DOMRpcRoutingTable.java:177)<span class="error">&#91;144:org.opendaylight.controller.sal-broker-impl:1.5.1.SNAPSHOT&#93;</span><br/> at org.opendaylight.controller.md.sal.dom.broker.impl.DOMRpcRouter.invokeRpc(DOMRpcRouter.java:102)<span class="error">&#91;144:org.opendaylight.controller.sal-broker-impl:1.5.1.SNAPSHOT&#93;</span><br/> at Proxy439bd690_1565_4870_9830_132c6fa3929d.invokeRpc(Unknown Source)<span class="error">&#91;:&#93;</span><br/> at Proxy7015ce82_5028_4759_a52a_1d18d2824edc.invokeRpc(Unknown Source)<span class="error">&#91;:&#93;</span><br/> at org.opendaylight.controller.md.sal.binding.impl.RpcServiceAdapter.invoke0(RpcServiceAdapter.java:65)<span class="error">&#91;146:org.opendaylight.controller.sal-binding-broker-impl:1.5.1.SNAPSHOT&#93;</span><br/> at org.opendaylight.controller.md.sal.binding.impl.RpcServiceAdapter.access$000(RpcServiceAdapter.java:43)<span class="error">&#91;146:org.opendaylight.controller.sal-binding-broker-impl:1.5.1.SNAPSHOT&#93;</span><br/> at org.opendaylight.controller.md.sal.binding.impl.RpcServiceAdapter$RpcInvocationStrategy.invoke(RpcServiceAdapter.java:159)<span class="error">&#91;146:org.opendaylight.controller.sal-binding-broker-impl:1.5.1.SNAPSHOT&#93;</span><br/> at org.opendaylight.controller.md.sal.binding.impl.RpcServiceAdapter.invoke(RpcServiceAdapter.java:96)<span class="error">&#91;146:org.opendaylight.controller.sal-binding-broker-impl:1.5.1.SNAPSHOT&#93;</span><br/> at com.sun.proxy.$Proxy129.releaseId(Unknown Source)<span class="error">&#91;290:org.opendaylight.genius.idmanager-api:0.2.1.SNAPSHOT&#93;</span><br/> at org.opendaylight.netvirt.aclservice.utils.AclServiceUtils.releaseId(AclServiceUtils.java:757)<span class="error">&#91;342:org.opendaylight.netvirt.aclservice-impl:0.4.1.SNAPSHOT&#93;</span><br/> at org.opendaylight.netvirt.aclservice.utils.AclServiceUtils.releaseAndRemoveFlowPriorityFromCache(AclServiceUtils.java:811)<span class="error">&#91;342:org.opendaylight.netvirt.aclservice-impl:0.4.1.SNAPSHOT&#93;</span><br/> at org.opendaylight.netvirt.aclservice.AbstractIngressAclServiceImpl.getIngressSpecificAclFlowPriority(AbstractIngressAclServiceImpl.java:408)<span class="error">&#91;342:org.opendaylight.netvirt.aclservice-impl:0.4.1.SNAPSHOT&#93;</span><br/> at org.opendaylight.netvirt.aclservice.StatefulIngressAclServiceImpl.syncSpecificAclFlow(StatefulIngressAclServiceImpl.java:104)<span class="error">&#91;342:org.opendaylight.netvirt.aclservice-impl:0.4.1.SNAPSHOT&#93;</span><br/> at org.opendaylight.netvirt.aclservice.AbstractIngressAclServiceImpl.programAceRule(AbstractIngressAclServiceImpl.java:225)<span class="error">&#91;342:org.opendaylight.netvirt.aclservice-impl:0.4.1.SNAPSHOT&#93;</span><br/> at org.opendaylight.netvirt.aclservice.AbstractIngressAclServiceImpl.programAclRules(AbstractIngressAclServiceImpl.java:196)<span class="error">&#91;342:org.opendaylight.netvirt.aclservice-impl:0.4.1.SNAPSHOT&#93;</span><br/> at org.opendaylight.netvirt.aclservice.AbstractAclServiceImpl.programAclWithAllowedAddress(AbstractAclServiceImpl.java:223)<span class="error">&#91;342:org.opendaylight.netvirt.aclservice-impl:0.4.1.SNAPSHOT&#93;</span><br/> at org.opendaylight.netvirt.aclservice.AbstractAclServiceImpl.removeAcl(AbstractAclServiceImpl.java:236)<span class="error">&#91;342:org.opendaylight.netvirt.aclservice-impl:0.4.1.SNAPSHOT&#93;</span><br/> at org.opendaylight.netvirt.aclservice.AclServiceManagerImpl.notify(AclServiceManagerImpl.java:60)<span class="error">&#91;342:org.opendaylight.netvirt.aclservice-impl:0.4.1.SNAPSHOT&#93;</span><br/> at org.opendaylight.netvirt.aclservice.listeners.AclInterfaceStateListener.remove(AclInterfaceStateListener.java:75)<span class="error">&#91;342:org.opendaylight.netvirt.aclservice-impl:0.4.1.SNAPSHOT&#93;</span><br/> at org.opendaylight.netvirt.aclservice.listeners.AclInterfaceStateListener.remove(AclInterfaceStateListener.java:34)<span class="error">&#91;342:org.opendaylight.netvirt.aclservice-impl:0.4.1.SNAPSHOT&#93;</span><br/> at org.opendaylight.genius.datastoreutils.AsyncDataTreeChangeListenerBase$DataTreeChangeHandler.run(AsyncDataTreeChangeListenerBase.java:149)<span class="error">&#91;291:org.opendaylight.genius.mdsalutil-api:0.2.1.SNAPSHOT&#93;</span><br/> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)<span class="error">&#91;:1.8.0_131&#93;</span><br/> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)<span class="error">&#91;:1.8.0_131&#93;</span><br/> at java.lang.Thread.run(Thread.java:748)<span class="error">&#91;:1.8.0_131&#93;</span></p> <p>Operating System: All<br/> Platform: All</p> NETVIRT-727

org.opendaylight.genius.idmanager.IdManagerException: Id Entries does not exist - ACLservice pool

Bug Resolved Done TARUN THAKUR Janki Chhatbar Tue, 13 Jun 2017 14:15:25 +0000 Mon, 30 Oct 2017 16:55:35 +0000 Sat, 12 Aug 2017 00:26:40 +0000 Carbon General 0 2 <p>##ID Allocation: Below 2 messages come from IDManager when ID is allocated to idkey.</p> <p>The newIdValues <span class="error">&#91;1011&#93;</span> for the idKey TCP_DESTINATION_1_0Ingress33216dc65c-dab2-44e4-a649-d5b3d094003c<br/> Updated id entry with idValues <span class="error">&#91;1011&#93;</span>, idKey TCP_DESTINATION_1_0Ingress33216dc65c-dab2-44e4-a649-d5b3d094003c, pool acl.flow.priorities.pool.123829938003760.243.PERMIT.169676308</p> <ol> <li> <ol> <li>ID Release failed: Below 3 messages come from IDManager when ID release is failed for idkey.</li> </ol> </li> </ol> <p>Releasing ID TCP_DESTINATION_1_0Ingress33216dc65c-dab2-44e4-a649-d5b3d094003c from pool acl.flow.priorities.pool.123829938003760.243.PERMIT<br/> releaseId failed: ReleaseIdInput [_idKey=TCP_DESTINATION_1_0Ingress33216dc65c-dab2-44e4-a649-d5b3d094003c, _poolName=acl.flow.priorities.pool.123829938003760.243.PERMIT, augmentation=[]]<br/> org.opendaylight.genius.idmanager.IdManagerException: Id Entries does not exist<br/> RPC Call to release Id TCP_DESTINATION_1_0Ingress33216dc65c-dab2-44e4-a649-d5b3d094003c with Key [RpcError [message=releaseId failed: ReleaseIdInput [_idKey=TCP_DESTINATION_1_0Ingress33216dc65c-dab2-44e4-a649-d5b3d094003c, _poolName=acl.flow.priorities.pool.123829938003760.243.PERMIT, augmentation=[]], severity=ERROR, errorType=APPLICATION, tag=operation-failed, applicationTag=null, info=null, cause=org.opendaylight.genius.idmanager.IdManagerException: Id Entries does not exist]] returned with Errors {}</p> <ol> <li> <ol> <li>ID Allocation: Below 2 messages come from IDManager when ID is allocated to idkey.</li> </ol> </li> </ol> <p>The newIdValues <span class="error">&#91;1000&#93;</span> for the idKey TCP_DESTINATION_1_0Ingress33216dc65c-dab2-44e4-a649-d5b3d094003c<br/> Updated id entry with idValues <span class="error">&#91;1000&#93;</span>, idKey TCP_DESTINATION_1_0Ingress33216dc65c-dab2-44e4-a649-d5b3d094003c, pool acl.flow.priorities.pool.123829938003760.243.PERMIT.169676308</p> <ol> <li> <ol> <li>ID Release Pass: Below 2 messages come from IDManager when ID release is successful for idkey.</li> </ol> </li> </ol> <p>Releasing ID TCP_DESTINATION_1_0Ingress33216dc65c-dab2-44e4-a649-d5b3d094003c from pool acl.flow.priorities.pool.123829938003760.243.PERMIT<br/> Updated id entry with idValues null, idKey TCP_DESTINATION_1_0Ingress33216dc65c-dab2-44e4-a649-d5b3d094003c, pool acl.flow.priorities.pool.123829938003760.243.PERMIT.169676308</p> <p>Same is the pattern for other keys (around 32 idKeys in karaf logs) as well.<br/> As per above pattern, I found that first time allocated ID is not getting released and fails, thereafter allocated IDs for idKey gets released successfully.</p> <p>This issue does not occur if VM or neutron port is deleted.</p> <p>But, it has been observed that when ovs (br-int on which neutron port is associated) is restarted and then after VM or neutron port is deleted, then releaseId fails.</p> <p>1. This issue is reproduced in dev environment locally on our laptop VMs.<br/> 2. Issue occurs only when OVS switch is restarted before deleting VM.<br/> 3. After switch restart, if VM is deleted, then ‘releaseId failed’ exception comes in the karaf logs. Root cause found for this issue is:</p> <p>3.a. After OVS restart, ACL pools are created again and performed through IdManager transaction. (NOTE: it consumes time)</p> <p>3.b. In parallel, control proceeds and other operations in ACL gets performed like pushing fixed rules, conntrack rules and then ACL rules.</p> <p>3.c. While ACL rules are being pushed, allocateId() method is getting called which allocates ID for ACL rule and IDManager’s allocateId() method tries to fetch pool in which it will allocate ID, but till this time ACL pool (as mentioned in above step 3.a) is not yet created, hence allocateId() method fails.</p> <p>3.d. Since allocateId() is failed and ID is not allocated, then later while VM deletion, releaseId() fails because it does not find ID in pool, which is very obvious.</p> <p>Issue is fixed and code change review is raised:<br/> <a href="https://git.opendaylight.org/gerrit/#/c/60012/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/60012/</a></p> <p>Seen in CSIT</p> <p>Seen in CSIT in latest build<br/> <a href="https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-newton-nodl-v2-upstream-stateful-nitrogen/123/odl1_karaf.log.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-newton-nodl-v2-upstream-stateful-nitrogen/123/odl1_karaf.log.gz</a></p> <p>2017-07-13 21:46:24,492 | ERROR | eChangeHandler-1 | IdManager | 228 - org.opendaylight.genius.idmanager-impl - 0.3.0.SNAPSHOT | releaseId failed: ReleaseIdInput [_idKey=TCP_DESTINATION_1_0Ingress1025410a80-dffd-40e2-99ed-1f50a6d20d53, _poolName=acl.flow.priorities.pool.159194546606218.243.PERMIT, augmentation=[]]<br/> ExpectedDataObjectNotFoundException{message=Expected to find data in CONFIGURATION at KeyedInstanceIdentifier</p> {targetType=interface org.opendaylight.yang.gen.v1.urn.opendaylight.genius.idmanager.rev160406.id.pools.IdPool, path=[org.opendaylight.yang.gen.v1.urn.opendaylight.genius.idmanager.rev160406.IdPools, org.opendaylight.yang.gen.v1.urn.opendaylight.genius.idmanager.rev160406.id.pools.IdPool[key=IdPoolKey [_poolName=acl.flow.priorities.pool.159194546606218.243.PERMIT]]]}, but there was none, errorList=[RpcError [message=Expected to find data in CONFIGURATION at KeyedInstanceIdentifier{targetType=interface org.opendaylight.yang.gen.v1.urn.opendaylight.genius.idmanager.rev160406.id.pools.IdPool, path=[org.opendaylight.yang.gen.v1.urn.opendaylight.genius.idmanager.rev160406.IdPools, org.opendaylight.yang.gen.v1.urn.opendaylight.genius.idmanager.rev160406.id.pools.IdPool[key=IdPoolKey [_poolName=acl.flow.priorities.pool.159194546606218.243.PERMIT]]]} <p>, but there was none, severity=ERROR, errorType=APPLICATION, tag=operation-failed, applicationTag=null, info=null, cause=null]]}<br/> at org.opendaylight.genius.datastoreutils.SingleTransactionDataBroker.syncRead(SingleTransactionDataBroker.java:112)<span class="error">&#91;237:org.opendaylight.genius.mdsalutil-api:0.3.0.SNAPSHOT&#93;</span><br/> at org.opendaylight.genius.datastoreutils.SingleTransactionDataBroker.syncRead(SingleTransactionDataBroker.java:100)<span class="error">&#91;237:org.opendaylight.genius.mdsalutil-api:0.3.0.SNAPSHOT&#93;</span><br/> at org.opendaylight.genius.idmanager.IdManager.releaseIdFromLocalPool(IdManager.java:634)<span class="error">&#91;228:org.opendaylight.genius.idmanager-impl:0.3.0.SNAPSHOT&#93;</span><br/> at org.opendaylight.genius.idmanager.IdManager.releaseId(IdManager.java:298)<span class="error">&#91;228:org.opendaylight.genius.idmanager-impl:0.3.0.SNAPSHOT&#93;</span><br/> at org.opendaylight.yangtools.yang.binding.util.RpcMethodInvokerWithInput.invokeOn(RpcMethodInvokerWithInput.java:31)<span class="error">&#91;285:org.opendaylight.mdsal.yang-binding:0.11.0.SNAPSHOT&#93;</span><br/> at org.opendaylight.yangtools.yang.binding.util.AbstractMappedRpcInvoker.invokeRpc(AbstractMappedRpcInvoker.java:52)<span class="error">&#91;285:org.opendaylight.mdsal.yang-binding:0.11.0.SNAPSHOT&#93;</span><br/> at org.opendaylight.controller.md.sal.binding.impl.BindingDOMRpcImplementationAdapter.invoke(BindingDOMRpcImplementationAdapter.java:83)<span class="error">&#91;198:org.opendaylight.controller.sal-binding-broker-impl:1.6.0.SNAPSHOT&#93;</span><br/> at org.opendaylight.controller.md.sal.binding.impl.BindingDOMRpcImplementationAdapter.invokeRpc(BindingDOMRpcImplementationAdapter.java:70)<span class="error">&#91;198:org.opendaylight.controller.sal-binding-broker-impl:1.6.0.SNAPSHOT&#93;</span><br/> at org.opendaylight.controller.md.sal.dom.broker.impl.GlobalDOMRpcRoutingTableEntry.invokeRpc(GlobalDOMRpcRoutingTableEntry.java:39)<span class="error">&#91;201:org.opendaylight.controller.sal-broker-impl:1.6.0.SNAPSHOT&#93;</span><br/> at org.opendaylight.controller.md.sal.dom.broker.impl.DOMRpcRoutingTable.invokeRpc(DOMRpcRoutingTable.java:177)<span class="error">&#91;201:org.opendaylight.controller.sal-broker-impl:1.6.0.SNAPSHOT&#93;</span><br/> at org.opendaylight.controller.md.sal.dom.broker.impl.DOMRpcRouter.invokeRpc(DOMRpcRouter.java:102)<span class="error">&#91;201:org.opendaylight.controller.sal-broker-impl:1.6.0.SNAPSHOT&#93;</span><br/> at Proxy8f3656f0_0eb6_4e36_97f4_3ff313b90630.invokeRpc(Unknown Source)<span class="error">&#91;:&#93;</span><br/> at Proxy38f9105e_0afd_412f_a32c_a40ea931afa0.invokeRpc(Unknown Source)<span class="error">&#91;:&#93;</span><br/> at org.opendaylight.controller.md.sal.binding.impl.RpcServiceAdapter.invoke0(RpcServiceAdapter.java:65)<span class="error">&#91;198:org.opendaylight.controller.sal-binding-broker-impl:1.6.0.SNAPSHOT&#93;</span><br/> at org.opendaylight.controller.md.sal.binding.impl.RpcServiceAdapter.access$000(RpcServiceAdapter.java:43)<span class="error">&#91;198:org.opendaylight.controller.sal-binding-broker-impl:1.6.0.SNAPSHOT&#93;</span><br/> at org.opendaylight.controller.md.sal.binding.impl.RpcServiceAdapter$RpcInvocationStrategy.invoke(RpcServiceAdapter.java:156)<span class="error">&#91;198:org.opendaylight.controller.sal-binding-broker-impl:1.6.0.SNAPSHOT&#93;</span><br/> at org.opendaylight.controller.md.sal.binding.impl.RpcServiceAdapter.invoke(RpcServiceAdapter.java:96)<span class="error">&#91;198:org.opendaylight.controller.sal-binding-broker-impl:1.6.0.SNAPSHOT&#93;</span><br/> at com.sun.proxy.$Proxy102.releaseId(Unknown Source)<span class="error">&#91;227:org.opendaylight.genius.idmanager-api:0.3.0.SNAPSHOT&#93;</span><br/> at org.opendaylight.netvirt.aclservice.utils.AclServiceUtils.releaseId(AclServiceUtils.java:783)<span class="error">&#91;291:org.opendaylight.netvirt.aclservice-impl:0.5.0.SNAPSHOT&#93;</span><br/> at org.opendaylight.netvirt.aclservice.utils.AclServiceUtils.releaseAndRemoveFlowPriorityFromCache(AclServiceUtils.java:839)<span class="error">&#91;291:org.opendaylight.netvirt.aclservice-impl:0.5.0.SNAPSHOT&#93;</span><br/> at org.opendaylight.netvirt.aclservice.AbstractIngressAclServiceImpl.getIngressSpecificAclFlowPriority(AbstractIngressAclServiceImpl.java:392)<span class="error">&#91;291:org.opendaylight.netvirt.aclservice-impl:0.5.0.SNAPSHOT&#93;</span><br/> at org.opendaylight.netvirt.aclservice.StatefulIngressAclServiceImpl.syncSpecificAclFlow(StatefulIngressAclServiceImpl.java:104)<span class="error">&#91;291:org.opendaylight.netvirt.aclservice-impl:0.5.0.SNAPSHOT&#93;</span><br/> at org.opendaylight.netvirt.aclservice.AbstractIngressAclServiceImpl.programAceRule(AbstractIngressAclServiceImpl.java:240)<span class="error">&#91;291:org.opendaylight.netvirt.aclservice-impl:0.5.0.SNAPSHOT&#93;</span><br/> at org.opendaylight.netvirt.aclservice.AbstractIngressAclServiceImpl.programAclRules(AbstractIngressAclServiceImpl.java:209)<span class="error">&#91;291:org.opendaylight.netvirt.aclservice-impl:0.5.0.SNAPSHOT&#93;</span><br/> at org.opendaylight.netvirt.aclservice.AbstractAclServiceImpl.updateCustomRules(AbstractAclServiceImpl.java:189)<span class="error">&#91;291:org.opendaylight.netvirt.aclservice-impl:0.5.0.SNAPSHOT&#93;</span><br/> at org.opendaylight.netvirt.aclservice.AbstractAclServiceImpl.processInterfaceUpdate(AbstractAclServiceImpl.java:171)<span class="error">&#91;291:org.opendaylight.netvirt.aclservice-impl:0.5.0.SNAPSHOT&#93;</span><br/> at org.opendaylight.netvirt.aclservice.AbstractAclServiceImpl.updateAcl(AbstractAclServiceImpl.java:142)<span class="error">&#91;291:org.opendaylight.netvirt.aclservice-impl:0.5.0.SNAPSHOT&#93;</span><br/> at org.opendaylight.netvirt.aclservice.AclServiceManagerImpl.notify(AclServiceManagerImpl.java:57)<span class="error">&#91;291:org.opendaylight.netvirt.aclservice-impl:0.5.0.SNAPSHOT&#93;</span><br/> at org.opendaylight.netvirt.aclservice.listeners.AclInterfaceListener.update(AclInterfaceListener.java:104)<span class="error">&#91;291:org.opendaylight.netvirt.aclservice-impl:0.5.0.SNAPSHOT&#93;</span><br/> at org.opendaylight.netvirt.aclservice.listeners.AclInterfaceListener.update(AclInterfaceListener.java:33)<span class="error">&#91;291:org.opendaylight.netvirt.aclservice-impl:0.5.0.SNAPSHOT&#93;</span><br/> at org.opendaylight.genius.datastoreutils.AsyncDataTreeChangeListenerBase$DataTreeChangeHandler.run(AsyncDataTreeChangeListenerBase.java:153)<span class="error">&#91;237:org.opendaylight.genius.mdsalutil-api:0.3.0.SNAPSHOT&#93;</span><br/> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)<span class="error">&#91;:1.8.0_131&#93;</span><br/> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)<span class="error">&#91;:1.8.0_131&#93;</span><br/> at java.lang.Thread.run(Thread.java:748)<span class="error">&#91;:1.8.0_131&#93;</span><br/> 2017-07-13 21:46:24,496 | WARN | eChangeHandler-1 | AclServiceUtils | 291 - org.opendaylight.netvirt.aclservice-impl - 0.5.0.SNAPSHOT | RPC Call to release Id TCP_DESTINATION_1_0Ingress1025410a80-dffd-40e2-99ed-1f50a6d20d53 with Key [RpcError [message=releaseId failed: ReleaseIdInput [_idKey=TCP_DESTINATION_1_0Ingress1025410a80-dffd-40e2-99ed-1f50a6d20d53, _poolName=acl.flow.priorities.pool.159194546606218.243.PERMIT, augmentation=[]], severity=ERROR, errorType=APPLICATION, tag=operation-failed, applicationTag=null, info=null, cause=ExpectedDataObjectNotFoundException{message=Expected to find data in CONFIGURATION at KeyedInstanceIdentifier</p> {targetType=interface org.opendaylight.yang.gen.v1.urn.opendaylight.genius.idmanager.rev160406.id.pools.IdPool, path=[org.opendaylight.yang.gen.v1.urn.opendaylight.genius.idmanager.rev160406.IdPools, org.opendaylight.yang.gen.v1.urn.opendaylight.genius.idmanager.rev160406.id.pools.IdPool[key=IdPoolKey [_poolName=acl.flow.priorities.pool.159194546606218.243.PERMIT]]]}, but there was none, errorList=[RpcError [message=Expected to find data in CONFIGURATION at KeyedInstanceIdentifier{targetType=interface org.opendaylight.yang.gen.v1.urn.opendaylight.genius.idmanager.rev160406.id.pools.IdPool, path=[org.opendaylight.yang.gen.v1.urn.opendaylight.genius.idmanager.rev160406.IdPools, org.opendaylight.yang.gen.v1.urn.opendaylight.genius.idmanager.rev160406.id.pools.IdPool[key=IdPoolKey [_poolName=acl.flow.priorities.pool.159194546606218.243.PERMIT]]]} <p>, but there was none, severity=ERROR, errorType=APPLICATION, tag=operation-failed, applicationTag=null, info=null, cause=null]]}], RpcError [message=Expected to find data in CONFIGURATION at KeyedInstanceIdentifier</p> {targetType=interface org.opendaylight.yang.gen.v1.urn.opendaylight.genius.idmanager.rev160406.id.pools.IdPool, path=[org.opendaylight.yang.gen.v1.urn.opendaylight.genius.idmanager.rev160406.IdPools, org.opendaylight.yang.gen.v1.urn.opendaylight.genius.idmanager.rev160406.id.pools.IdPool[key=IdPoolKey [_poolName=acl.flow.priorities.pool.159194546606218.243.PERMIT]]]} <p>, but there was none, severity=ERROR, errorType=APPLICATION, tag=operation-failed, applicationTag=null, info=null, cause=null]] returned with Errors {}</p> <p>Different keyword but root cause could be same.</p> <p>Seen in CSIT in latest build<br/> <a href="https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-newton-nodl-v2-upstream-stateful-nitrogen/123/odl1_karaf.log.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-newton-nodl-v2-upstream-stateful-nitrogen/123/odl1_karaf.log.gz</a></p> <p>2017-07-13 21:46:24,492 | ERROR | eChangeHandler-1 | IdManager | 228 - org.opendaylight.genius.idmanager-impl - 0.3.0.SNAPSHOT | releaseId failed: ReleaseIdInput [_idKey=TCP_DESTINATION_1_0Ingress1025410a80-dffd-40e2-99ed-1f50a6d20d53, _poolName=acl.flow.priorities.pool.159194546606218.243.PERMIT, augmentation=[]]<br/> ExpectedDataObjectNotFoundException{message=Expected to find data in CONFIGURATION at KeyedInstanceIdentifier</p> {targetType=interface org.opendaylight.yang.gen.v1.urn.opendaylight.genius.idmanager.rev160406.id.pools.IdPool, path=[org.opendaylight.yang.gen.v1.urn.opendaylight.genius.idmanager.rev160406.IdPools, org.opendaylight.yang.gen.v1.urn.opendaylight.genius.idmanager.rev160406.id.pools.IdPool[key=IdPoolKey [_poolName=acl.flow.priorities.pool.159194546606218.243.PERMIT]]]}, but there was none, errorList=[RpcError [message=Expected to find data in CONFIGURATION at KeyedInstanceIdentifier{targetType=interface org.opendaylight.yang.gen.v1.urn.opendaylight.genius.idmanager.rev160406.id.pools.IdPool, path=[org.opendaylight.yang.gen.v1.urn.opendaylight.genius.idmanager.rev160406.IdPools, org.opendaylight.yang.gen.v1.urn.opendaylight.genius.idmanager.rev160406.id.pools.IdPool[key=IdPoolKey [_poolName=acl.flow.priorities.pool.159194546606218.243.PERMIT]]]} <p>, but there was none, severity=ERROR, errorType=APPLICATION, tag=operation-failed, applicationTag=null, info=null, cause=null]]}<br/> at org.opendaylight.genius.datastoreutils.SingleTransactionDataBroker.syncRead(SingleTransactionDataBroker.java:112)<span class="error">&#91;237:org.opendaylight.genius.mdsalutil-api:0.3.0.SNAPSHOT&#93;</span><br/> at org.opendaylight.genius.datastoreutils.SingleTransactionDataBroker.syncRead(SingleTransactionDataBroker.java:100)<span class="error">&#91;237:org.opendaylight.genius.mdsalutil-api:0.3.0.SNAPSHOT&#93;</span><br/> at org.opendaylight.genius.idmanager.IdManager.releaseIdFromLocalPool(IdManager.java:634)<span class="error">&#91;228:org.opendaylight.genius.idmanager-impl:0.3.0.SNAPSHOT&#93;</span><br/> at org.opendaylight.genius.idmanager.IdManager.releaseId(IdManager.java:298)<span class="error">&#91;228:org.opendaylight.genius.idmanager-impl:0.3.0.SNAPSHOT&#93;</span><br/> at org.opendaylight.yangtools.yang.binding.util.RpcMethodInvokerWithInput.invokeOn(RpcMethodInvokerWithInput.java:31)<span class="error">&#91;285:org.opendaylight.mdsal.yang-binding:0.11.0.SNAPSHOT&#93;</span><br/> at org.opendaylight.yangtools.yang.binding.util.AbstractMappedRpcInvoker.invokeRpc(AbstractMappedRpcInvoker.java:52)<span class="error">&#91;285:org.opendaylight.mdsal.yang-binding:0.11.0.SNAPSHOT&#93;</span><br/> at org.opendaylight.controller.md.sal.binding.impl.BindingDOMRpcImplementationAdapter.invoke(BindingDOMRpcImplementationAdapter.java:83)<span class="error">&#91;198:org.opendaylight.controller.sal-binding-broker-impl:1.6.0.SNAPSHOT&#93;</span><br/> at org.opendaylight.controller.md.sal.binding.impl.BindingDOMRpcImplementationAdapter.invokeRpc(BindingDOMRpcImplementationAdapter.java:70)<span class="error">&#91;198:org.opendaylight.controller.sal-binding-broker-impl:1.6.0.SNAPSHOT&#93;</span><br/> at org.opendaylight.controller.md.sal.dom.broker.impl.GlobalDOMRpcRoutingTableEntry.invokeRpc(GlobalDOMRpcRoutingTableEntry.java:39)<span class="error">&#91;201:org.opendaylight.controller.sal-broker-impl:1.6.0.SNAPSHOT&#93;</span><br/> at org.opendaylight.controller.md.sal.dom.broker.impl.DOMRpcRoutingTable.invokeRpc(DOMRpcRoutingTable.java:177)<span class="error">&#91;201:org.opendaylight.controller.sal-broker-impl:1.6.0.SNAPSHOT&#93;</span><br/> at org.opendaylight.controller.md.sal.dom.broker.impl.DOMRpcRouter.invokeRpc(DOMRpcRouter.java:102)<span class="error">&#91;201:org.opendaylight.controller.sal-broker-impl:1.6.0.SNAPSHOT&#93;</span><br/> at Proxy8f3656f0_0eb6_4e36_97f4_3ff313b90630.invokeRpc(Unknown Source)<span class="error">&#91;:&#93;</span><br/> at Proxy38f9105e_0afd_412f_a32c_a40ea931afa0.invokeRpc(Unknown Source)<span class="error">&#91;:&#93;</span><br/> at org.opendaylight.controller.md.sal.binding.impl.RpcServiceAdapter.invoke0(RpcServiceAdapter.java:65)<span class="error">&#91;198:org.opendaylight.controller.sal-binding-broker-impl:1.6.0.SNAPSHOT&#93;</span><br/> at org.opendaylight.controller.md.sal.binding.impl.RpcServiceAdapter.access$000(RpcServiceAdapter.java:43)<span class="error">&#91;198:org.opendaylight.controller.sal-binding-broker-impl:1.6.0.SNAPSHOT&#93;</span><br/> at org.opendaylight.controller.md.sal.binding.impl.RpcServiceAdapter$RpcInvocationStrategy.invoke(RpcServiceAdapter.java:156)<span class="error">&#91;198:org.opendaylight.controller.sal-binding-broker-impl:1.6.0.SNAPSHOT&#93;</span><br/> at org.opendaylight.controller.md.sal.binding.impl.RpcServiceAdapter.invoke(RpcServiceAdapter.java:96)<span class="error">&#91;198:org.opendaylight.controller.sal-binding-broker-impl:1.6.0.SNAPSHOT&#93;</span><br/> at com.sun.proxy.$Proxy102.releaseId(Unknown Source)<span class="error">&#91;227:org.opendaylight.genius.idmanager-api:0.3.0.SNAPSHOT&#93;</span><br/> at org.opendaylight.netvirt.aclservice.utils.AclServiceUtils.releaseId(AclServiceUtils.java:783)<span class="error">&#91;291:org.opendaylight.netvirt.aclservice-impl:0.5.0.SNAPSHOT&#93;</span><br/> at org.opendaylight.netvirt.aclservice.utils.AclServiceUtils.releaseAndRemoveFlowPriorityFromCache(AclServiceUtils.java:839)<span class="error">&#91;291:org.opendaylight.netvirt.aclservice-impl:0.5.0.SNAPSHOT&#93;</span><br/> at org.opendaylight.netvirt.aclservice.AbstractIngressAclServiceImpl.getIngressSpecificAclFlowPriority(AbstractIngressAclServiceImpl.java:392)<span class="error">&#91;291:org.opendaylight.netvirt.aclservice-impl:0.5.0.SNAPSHOT&#93;</span><br/> at org.opendaylight.netvirt.aclservice.StatefulIngressAclServiceImpl.syncSpecificAclFlow(StatefulIngressAclServiceImpl.java:104)<span class="error">&#91;291:org.opendaylight.netvirt.aclservice-impl:0.5.0.SNAPSHOT&#93;</span><br/> at org.opendaylight.netvirt.aclservice.AbstractIngressAclServiceImpl.programAceRule(AbstractIngressAclServiceImpl.java:240)<span class="error">&#91;291:org.opendaylight.netvirt.aclservice-impl:0.5.0.SNAPSHOT&#93;</span><br/> at org.opendaylight.netvirt.aclservice.AbstractIngressAclServiceImpl.programAclRules(AbstractIngressAclServiceImpl.java:209)<span class="error">&#91;291:org.opendaylight.netvirt.aclservice-impl:0.5.0.SNAPSHOT&#93;</span><br/> at org.opendaylight.netvirt.aclservice.AbstractAclServiceImpl.updateCustomRules(AbstractAclServiceImpl.java:189)<span class="error">&#91;291:org.opendaylight.netvirt.aclservice-impl:0.5.0.SNAPSHOT&#93;</span><br/> at org.opendaylight.netvirt.aclservice.AbstractAclServiceImpl.processInterfaceUpdate(AbstractAclServiceImpl.java:171)<span class="error">&#91;291:org.opendaylight.netvirt.aclservice-impl:0.5.0.SNAPSHOT&#93;</span><br/> at org.opendaylight.netvirt.aclservice.AbstractAclServiceImpl.updateAcl(AbstractAclServiceImpl.java:142)<span class="error">&#91;291:org.opendaylight.netvirt.aclservice-impl:0.5.0.SNAPSHOT&#93;</span><br/> at org.opendaylight.netvirt.aclservice.AclServiceManagerImpl.notify(AclServiceManagerImpl.java:57)<span class="error">&#91;291:org.opendaylight.netvirt.aclservice-impl:0.5.0.SNAPSHOT&#93;</span><br/> at org.opendaylight.netvirt.aclservice.listeners.AclInterfaceListener.update(AclInterfaceListener.java:104)<span class="error">&#91;291:org.opendaylight.netvirt.aclservice-impl:0.5.0.SNAPSHOT&#93;</span><br/> at org.opendaylight.netvirt.aclservice.listeners.AclInterfaceListener.update(AclInterfaceListener.java:33)<span class="error">&#91;291:org.opendaylight.netvirt.aclservice-impl:0.5.0.SNAPSHOT&#93;</span><br/> at org.opendaylight.genius.datastoreutils.AsyncDataTreeChangeListenerBase$DataTreeChangeHandler.run(AsyncDataTreeChangeListenerBase.java:153)<span class="error">&#91;237:org.opendaylight.genius.mdsalutil-api:0.3.0.SNAPSHOT&#93;</span><br/> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)<span class="error">&#91;:1.8.0_131&#93;</span><br/> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)<span class="error">&#91;:1.8.0_131&#93;</span><br/> at java.lang.Thread.run(Thread.java:748)<span class="error">&#91;:1.8.0_131&#93;</span><br/> 2017-07-13 21:46:24,496 | WARN | eChangeHandler-1 | AclServiceUtils | 291 - org.opendaylight.netvirt.aclservice-impl - 0.5.0.SNAPSHOT | RPC Call to release Id TCP_DESTINATION_1_0Ingress1025410a80-dffd-40e2-99ed-1f50a6d20d53 with Key [RpcError [message=releaseId failed: ReleaseIdInput [_idKey=TCP_DESTINATION_1_0Ingress1025410a80-dffd-40e2-99ed-1f50a6d20d53, _poolName=acl.flow.priorities.pool.159194546606218.243.PERMIT, augmentation=[]], severity=ERROR, errorType=APPLICATION, tag=operation-failed, applicationTag=null, info=null, cause=ExpectedDataObjectNotFoundException{message=Expected to find data in CONFIGURATION at KeyedInstanceIdentifier</p> {targetType=interface org.opendaylight.yang.gen.v1.urn.opendaylight.genius.idmanager.rev160406.id.pools.IdPool, path=[org.opendaylight.yang.gen.v1.urn.opendaylight.genius.idmanager.rev160406.IdPools, org.opendaylight.yang.gen.v1.urn.opendaylight.genius.idmanager.rev160406.id.pools.IdPool[key=IdPoolKey [_poolName=acl.flow.priorities.pool.159194546606218.243.PERMIT]]]}, but there was none, errorList=[RpcError [message=Expected to find data in CONFIGURATION at KeyedInstanceIdentifier{targetType=interface org.opendaylight.yang.gen.v1.urn.opendaylight.genius.idmanager.rev160406.id.pools.IdPool, path=[org.opendaylight.yang.gen.v1.urn.opendaylight.genius.idmanager.rev160406.IdPools, org.opendaylight.yang.gen.v1.urn.opendaylight.genius.idmanager.rev160406.id.pools.IdPool[key=IdPoolKey [_poolName=acl.flow.priorities.pool.159194546606218.243.PERMIT]]]} <p>, but there was none, severity=ERROR, errorType=APPLICATION, tag=operation-failed, applicationTag=null, info=null, cause=null]]}], RpcError [message=Expected to find data in CONFIGURATION at KeyedInstanceIdentifier</p> {targetType=interface org.opendaylight.yang.gen.v1.urn.opendaylight.genius.idmanager.rev160406.id.pools.IdPool, path=[org.opendaylight.yang.gen.v1.urn.opendaylight.genius.idmanager.rev160406.IdPools, org.opendaylight.yang.gen.v1.urn.opendaylight.genius.idmanager.rev160406.id.pools.IdPool[key=IdPoolKey [_poolName=acl.flow.priorities.pool.159194546606218.243.PERMIT]]]} <p>, but there was none, severity=ERROR, errorType=APPLICATION, tag=operation-failed, applicationTag=null, info=null, cause=null]] returned with Errors {}</p> <p>Different keyword but root cause could be same.</p> <p><a href="https://git.opendaylight.org/gerrit/60012" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/60012</a></p> Development External issue ID 8683 External issue URL Rank 0|i01tl3: Status Whiteboard csit:exception