Exception in AclInterfaceCacheUtil

Smaller description:
---------------------
When a neutron subnet is created, the following exception is seen in ACLservice.

2016-09-01 18:32:44,258 | WARN | on-dispatcher-39 | VpnInterfaceManager | 341 - org.opendaylight.netvirt.vpnmanager-impl - 0.4.0.SNAPSHOT | VPN interface a076de9d-ca71-46c8-b789-0887a801bfe1 was unavailable in operational data store to handle remove event
Exception in thread "pool-43-thread-1" java.lang.NullPointerException
at org.opendaylight.netvirt.aclservice.api.utils.AclInterfaceCacheUtil.removeAclInterfaceFromCache(AclInterfaceCacheUtil.java:22)
at org.opendaylight.netvirt.aclservice.listeners.AclInterfaceListener.remove(AclInterfaceListener.java:69)
at org.opendaylight.netvirt.aclservice.listeners.AclInterfaceListener.remove(AclInterfaceListener.java:33)
at org.opendaylight.genius.datastoreutils.AsyncDataTreeChangeListenerBase$DataTreeChangeHandler.run(AsyncDataTreeChangeListenerBase.java:117)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2016-09-01 18:32:44,618 | INFO | on-dispatcher-40 | NeutronPortChangeListener |

Longer description:
=============================================================================
In my all-in-one devstack setup, I've noticed the following exception in dhcp-agent logs. Basically it implies that dhcp-agent could not find the dhcp tap interface, so its re-creating the dhcp-interface (which is failing for some reason). This operation goes into indefinite loop and we see the above exception from ACLService.

After re-starting the dhcp-agent, the problem disappeared.

2016-09-01 13:41:37.112 ERROR neutron.agent.dhcp.agent [-] Unable to enable dhcp for 4f27ed6d-8daa-467e-aa45-0ab92b2d5aac.
2016-09-01 13:41:37.112 TRACE neutron.agent.dhcp.agent Traceback (most recent call last):
2016-09-01 13:41:37.112 TRACE neutron.agent.dhcp.agent File "/opt/stack/neutron/neutron/agent/dhcp/agent.py", line 115, in call_driver
2016-09-01 13:41:37.112 TRACE neutron.agent.dhcp.agent getattr(driver, action)(**action_kwargs)
2016-09-01 13:41:37.112 TRACE neutron.agent.dhcp.agent File "/opt/stack/neutron/neutron/agent/linux/dhcp.py", line 211, in enable
2016-09-01 13:41:37.112 TRACE neutron.agent.dhcp.agent interface_name = self.device_manager.setup(self.network)
2016-09-01 13:41:37.112 TRACE neutron.agent.dhcp.agent File "/opt/stack/neutron/neutron/agent/linux/dhcp.py", line 1240, in setup
2016-09-01 13:41:37.112 TRACE neutron.agent.dhcp.agent self.plugin.release_dhcp_port(network.id, port.device_id)
2016-09-01 13:41:37.112 TRACE neutron.agent.dhcp.agent File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in _exit_
2016-09-01 13:41:37.112 TRACE neutron.agent.dhcp.agent self.force_reraise()
2016-09-01 13:41:37.112 TRACE neutron.agent.dhcp.agent File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-09-01 13:41:37.112 TRACE neutron.agent.dhcp.agent six.reraise(self.type_, self.value, self.tb)
2016-09-01 13:41:37.112 TRACE neutron.agent.dhcp.agent File "/opt/stack/neutron/neutron/agent/linux/dhcp.py", line 1234, in setup
2016-09-01 13:41:37.112 TRACE neutron.agent.dhcp.agent mtu=network.get('mtu'))
2016-09-01 13:41:37.112 TRACE neutron.agent.dhcp.agent File "/opt/stack/neutron/neutron/agent/linux/interface.py", line 232, in plug
2016-09-01 13:41:37.112 TRACE neutron.agent.dhcp.agent bridge, namespace, prefix, mtu)
2016-09-01 13:41:37.112 TRACE neutron.agent.dhcp.agent File "/opt/stack/neutron/neutron/agent/linux/interface.py", line 325, in plug_new
2016-09-01 13:41:37.112 TRACE neutron.agent.dhcp.agent ns_dev.link.set_address(mac_address)
2016-09-01 13:41:37.112 TRACE neutron.agent.dhcp.agent File "/opt/stack/neutron/neutron/agent/linux/ip_lib.py", line 496, in set_address
2016-09-01 13:41:37.112 TRACE neutron.agent.dhcp.agent self._as_root([], ('set', self.name, 'address', mac_address))
2016-09-01 13:41:37.112 TRACE neutron.agent.dhcp.agent File "/opt/stack/neutron/neutron/agent/linux/ip_lib.py", line 362, in _as_root
2016-09-01 13:41:37.112 TRACE neutron.agent.dhcp.agent use_root_namespace=use_root_namespace)
2016-09-01 13:41:37.112 TRACE neutron.agent.dhcp.agent File "/opt/stack/neutron/neutron/agent/linux/ip_lib.py", line 95, in _as_root
2016-09-01 13:41:37.112 TRACE neutron.agent.dhcp.agent log_fail_as_error=self.log_fail_as_error)
2016-09-01 13:41:37.112 TRACE neutron.agent.dhcp.agent File "/opt/stack/neutron/neutron/agent/linux/ip_lib.py", line 104, in _execute
2016-09-01 13:41:37.112 TRACE neutron.agent.dhcp.agent log_fail_as_error=log_fail_as_error)
2016-09-01 13:41:37.112 TRACE neutron.agent.dhcp.agent File "/opt/stack/neutron/neutron/agent/linux/utils.py", line 138, in execute
2016-09-01 13:41:37.112 TRACE neutron.agent.dhcp.agent raise RuntimeError(msg)
2016-09-01 13:41:37.112 TRACE neutron.agent.dhcp.agent RuntimeError: Exit code: 1; Stdin: ; Stdout: ; Stderr: Cannot find device "tap8fc0a982-68"

2016-09-01 13:25:34.025 ERROR oslo_messaging.rpc.server [req-9a990c4a-bafc-4014-93bb-7746417fa61f None None] Exception during message handling
2016-09-01 13:25:34.025 TRACE oslo_messaging.rpc.server Traceback (most recent call last):
2016-09-01 13:25:34.025 TRACE oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 133, in _process_incoming
2016-09-01 13:25:34.025 TRACE oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2016-09-01 13:25:34.025 TRACE oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 150, in dispatch
2016-09-01 13:25:34.025 TRACE oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2016-09-01 13:25:34.025 TRACE oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 121, in _do_dispatch
2016-09-01 13:25:34.025 TRACE oslo_messaging.rpc.server result = func(ctxt, **new_args)
2016-09-01 13:25:34.025 TRACE oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 151, in wrapper
2016-09-01 13:25:34.025 TRACE oslo_messaging.rpc.server ectxt.value = e.inner_exc
2016-09-01 13:25:34.025 TRACE oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in _exit_
2016-09-01 13:25:34.025 TRACE oslo_messaging.rpc.server self.force_reraise()
2016-09-01 13:25:34.025 TRACE oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-09-01 13:25:34.025 TRACE oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2016-09-01 13:25:34.025 TRACE oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 139, in wrapper
2016-09-01 13:25:34.025 TRACE oslo_messaging.rpc.server return f(*args, **kwargs)
2016-09-01 13:25:34.025 TRACE oslo_messaging.rpc.server File "/opt/stack/neutron/neutron/db/api.py", line 74, in wrapped
2016-09-01 13:25:34.025 TRACE oslo_messaging.rpc.server traceback.format_exc())
2016-09-01 13:25:34.025 TRACE oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in _exit_
2016-09-01 13:25:34.025 TRACE oslo_messaging.rpc.server self.force_reraise()
2016-09-01 13:25:34.025 TRACE oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-09-01 13:25:34.025 TRACE oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2016-09-01 13:25:34.025 TRACE oslo_messaging.rpc.server File "/opt/stack/neutron/neutron/db/api.py", line 69, in wrapped
2016-09-01 13:25:34.025 TRACE oslo_messaging.rpc.server return f(*args, **kwargs)
2016-09-01 13:25:34.025 TRACE oslo_messaging.rpc.server File "/opt/stack/neutron/neutron/quota/resource_registry.py", line 92, in wrapper
2016-09-01 13:25:34.025 TRACE oslo_messaging.rpc.server ret_val = f(_self, context, *args, **kwargs)
2016-09-01 13:25:34.025 TRACE oslo_messaging.rpc.server File "/opt/stack/neutron/neutron/api/rpc/handlers/dhcp_rpc.py", line 254, in create_dhcp_port
2016-09-01 13:25:34.025 TRACE oslo_messaging.rpc.server return self._port_action(plugin, context, port, 'create_port')
2016-09-01 13:25:34.025 TRACE oslo_messaging.rpc.server File "/opt/stack/neutron/neutron/api/rpc/handlers/dhcp_rpc.py", line 98, in _port_action
2016-09-01 13:25:34.025 TRACE oslo_messaging.rpc.server return p_utils.create_port(plugin, context, port)
2016-09-01 13:25:34.025 TRACE oslo_messaging.rpc.server File "/opt/stack/neutron/neutron/plugins/common/utils.py", line 197, in create_port
2016-09-01 13:25:34.025 TRACE oslo_messaging.rpc.server return core_plugin.create_port(context,

)
2016-09-01 13:25:34.025 TRACE oslo_messaging.rpc.server File "/opt/stack/neutron/neutron/common/utils.py", line 615, in inner
2016-09-01 13:25:34.025 TRACE oslo_messaging.rpc.server return f(self, context, *args, **kwargs)
2016-09-01 13:25:34.025 TRACE oslo_messaging.rpc.server File "/opt/stack/neutron/neutron/plugins/ml2/plugin.py", line 1248, in create_port
2016-09-01 13:25:34.025 TRACE oslo_messaging.rpc.server self.delete_port(context, result['id'], l3_port_check=False)
2016-09-01 13:25:34.025 TRACE oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in _exit_
2016-09-01 13:25:34.025 TRACE oslo_messaging.rpc.server self.force_reraise()
2016-09-01 13:25:34.025 TRACE oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-09-01 13:25:34.025 TRACE oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2016-09-01 13:25:34.025 TRACE oslo_messaging.rpc.server File "/opt/stack/neutron/neutron/plugins/ml2/plugin.py", line 1243, in create_port
2016-09-01 13:25:34.025 TRACE oslo_messaging.rpc.server self.mechanism_manager.create_port_postcommit(mech_context)
2016-09-01 13:25:34.025 TRACE oslo_messaging.rpc.server File "/opt/stack/neutron/neutron/plugins/ml2/managers.py", line 647, in create_port_postcommit
2016-09-01 13:25:34.025 TRACE oslo_messaging.rpc.server self._call_on_drivers("create_port_postcommit", context)
2016-09-01 13:25:34.025 TRACE oslo_messaging.rpc.server File "/opt/stack/neutron/neutron/plugins/ml2/managers.py", line 427, in _call_on_drivers
2016-09-01 13:25:34.025 TRACE oslo_messaging.rpc.server method=method_name
2016-09-01 13:25:34.025 TRACE oslo_messaging.rpc.server MechanismDriverError: create_port_postcommit failed.
2016-09-01 13:25:34.025 TRACE oslo_messaging.rpc.server

[vagrant@f23-aione-master-odl devstack]$ neutron port-list
------------------------------------------------------------------------------------------------------------------------------------------+

------------------------------------------------------------------------------------------------------------------------------------------+

------------------------------------------------------------------------------------------------------------------------------------------+
[vagrant@f23-aione-master-odl devstack]$ neutron port-list
------------------------------------------------------------------------------------------------------------------------------------------+

------------------------------------------------------------------------------------------------------------------------------------------+

------------------------------------------------------------------------------------------------------------------------------------------+
[vagrant@f23-aione-master-odl devstack]$ neutron port-list
------------------------------------------------------------------------------------------------------------------------------------------+

------------------------------------------------------------------------------------------------------------------------------------------+

------------------------------------------------------------------------------------------------------------------------------------------+
[vagrant@f23-aione-master-odl devstack]$
[vagrant@f23-aione-master-odl devstack]$ neutron port-list
------------------------------------------------------------------------------------------------------------------------------------------+

------------------------------------------------------------------------------------------------------------------------------------------+

------------------------------------------------------------------------------------------------------------------------------------------+
[vagrant@f23-aione-master-odl devstack]$ neutron port-list
------------------------------------------------------------------------------------------------------------------------------------------+

------------------------------------------------------------------------------------------------------------------------------------------+

------------------------------------------------------------------------------------------------------------------------------------------+
[vagrant@f23-aione-master-odl devstack]$ neutron port-list
------------------------------------------------------------------------------------------------------------------------------------------+

------------------------------------------------------------------------------------------------------------------------------------------+

------------------------------------------------------------------------------------------------------------------------------------------+