[CONTROLLER-1702] Performance: write-transactions is unable to fill the initial item list within 5 seconds

Tue, 25 Jul 2023 08:24:41 +0000

This is a Carbon pre-SR1 bug not currently visible on RelEng, as it is hidden by ~~CONTROLLER-1687~~.

Write-transactions is an RPC used to test clustering under load. It starts by creating a list of integer leaf items and then doing random updates to it. The initial size is 2^19 items, written in a single transaction.

The Robot tests currently wait for 5 seconds before affecting the cluster, so the transaction has time to be committed. Sandbox test run with fix of ~~CONTROLLER-1687~~ shows that sometimes the test passes [0], but sometimes it fails by encountering [1] RequestTimeoutException just few seconds before 120 second limit.
But the response states the wait was over 120 seconds, and the karaf.log [2] confirms the exception happens in the initial item list filling transaction:

{message=canCommit encountered an unexpected failure, errorList=[RpcError [message=canCommit encountered an unexpected failure, severity=ERROR, errorType=APPLICATION, tag=operation-failed, applicationTag=null, info=null, cause=org.opendaylight.controller.cluster.access.client.RequestTimeoutException: Timed out after 120029545279ns]]}

This might be a real performance bug, or it might be expected (as the suite performs reboots between tests, so the config datastore is not warmed up yet). If this is expected, we will increase the sleep time in Robot.

[0] https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-only-carbon/4/log.html.gz#s1-s14-t1
[1] https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-only-carbon/4/log.html.gz#s1-s14-t3-k2-k15-k1-k1-k1-k1-k1-k1-k2-k1-k1-k1
[2] https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-only-carbon/4/odl1_karaf.log.gz

OpenDaylight JIRA

[CONTROLLER-1702] Performance: write-transactions is unable to fill the initial item list within 5 seconds