[CONTROLLER-1700] Timeout waiting for task from writer started after heal after long isolation

Blocks — Tue, 25 Jul 2023 08:24:41 +0000

This symptom is very similar to ~~CONTROLLER-1674~~ but happens in a different scenario.

Still three writers (module-based shards, tell-based protocol) and leader is isolated, for more than request timeout. The isolated writer fails (with TimeoutException instead of RequestTimeoutException, that might be a separate bug or a cause of this one). Then the member is rejoined and it is verified each shard has a Leader and two Followers.

At the end of the scenario, we start a writer on the rejoined node, and we expect it to finish writing without errors. Instead finishing after 67 seconds, TimeoutException is seen. The other two writers finish correctly.

In a recent Sandbox test [0] (which failed to upload archive with logs) the response starts with:
{"errors":{"error":[{"error-type":"application","error-tag":"operation-failed","error-message":"Unexpected-exception","error-info":"java.util.concurrent.TimeoutException: Timeout waiting for task.\n\tat com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:269)\n\tat com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:96)\n\tat com.google.common.util.concurrent.ForwardingFuture.get(ForwardingFuture.java:69)\n\tat org.opendaylight.yangtools.util.concurrent.MappingCheckedFuture.get(MappingCheckedFuture.java:85)\n\tat com.google.common.util.concurrent.AbstractCheckedFuture.checkedGet(AbstractCheckedFuture.java:107)\n\tat org.opendaylight.controller.clustering.it.provider.impl.WriteTransactionsHandler.ensureListExists(WriteTransactionsHandler.java:146)\n\tat org.opendaylight.controller.clustering.it.provider.impl.WriteTransactionsHandler.start(WriteTransactionsHandler.java:126)\n\tat org.opendaylight.controller.clustering.it.provider.MdsalLowLevelTestProvider.writeTransactions(MdsalLowLevelTestProvider.java:250)\n\tat org.opendaylight.yangtools.yang.binding.util.RpcMethodInvokerWithInput.invokeOn(RpcMethodInvokerWithInput.java:31)\n\tat

[0] https://jenkins.opendaylight.org/sandbox/job/controller-csit-3node-clustering-only-carbon/6/robot/report/log.html#s1-s10-t3-k2-k25-k1-k8

OpenDaylight JIRA

[CONTROLLER-1700] Timeout waiting for task from writer started after heal after long isolation