[OPNFLWPLUG-859] Internal SalSevice queue gets full and ODL does not perform any more Openflow actions in the switch and does not release the mastership

Mon, 27 Sep 2021 09:02:00 +0000

After running ODL for few days, it seems some internal queue gets fulls and SalService does not allow to perform any action on the switch. The connectivity with the switch is established and the hello messages from the switch are returned properly.

Restarting the switch does not solve the problem, the controller that holds the mastership does not free the cluster singleton service. The rest of the controller that holds "candidate" status on the cluster singleton (slave mode in the switch) release the singleton service.

The problem was solved only after restarting the controller that holds the mastership to the switch.

I noticed exceptions that may point to the source of the problem. Basically, following code does not return a requestContext . (requestContext is always null).

final RequestContext<O> requestContext = requestContextStack.createRequestContext();
if (requestContext == null)

{ LOG.trace("Request context refused."); getMessageSpy().spyMessage(AbstractService.class, MessageSpy.STATISTIC_GROUP.TO_SWITCH_DISREGARDED); return failedFuture(); }

And following logs points that the queue is full.

The log "Device queue org.opendaylight.openflowplugin.i mpl.rpc.RpcContextImpl@3a502d68 at capacity" is returned because following code cannot acquire the lock which is a semaphore.

public <T> RequestContext<T> createRequestContext() {
if (!tracker.tryAcquire()) {
LOG.trace("Device queue {} at capacity", this);
return null;
} else {
LOG.trace("Acquired semaphore for {}, available permits:{} ", nodeInstanceIdentifier.getKey().getId().getValue(), tracker.availablePermits());
}

OpenDaylight JIRA

[OPNFLWPLUG-859] Internal SalSevice queue gets full and ODL does not perform any more Openflow actions in the switch and does not release the mastership