[CONTROLLER-1881] DeadTransactionException failure in ELM longevity test

Sun, 31 May 2020 06:21:47 +0000

This is occurring consistently, eg https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/controller-csit-3node-ddb-expl-lead-movement-longevity-only-neon/21/robot-plugin/log.html.gz

In the iteration that failed, odl2 was the shard leader ${idx_from} and the node picked to generate transactions ${idx_trans}, odl1 was the node to make the new leader ${idx_to}.

The odl-mdsal-lowlevel-control:write-transactions RPC to odl2 failed in WriteTransactionsHandler.start when attempting to write the initial top level nodes, using standalone transactions.

In odl2 karaf log, we see:

2019-01-06T21:32:51,827 | WARN  | opendaylight-cluster-data-shard-dispatcher-27 | AbstractFrontendHistory          | 226 - org.opendaylight.controller.sal-distributed-datastore - 1.9.0.SNAPSHOT | member-2-shard-default-config: Request CommitLocalTransactionRequest{target=member-2-datastore-config-fe-0-txn-1546-0, sequence=0, replyTo=Actor[akka://opendaylight-cluster-data/user/$a#-1325814690], coordinated=false} is contained purged transactions [[0..1546]]

2019-01-06T21:32:51,887 | WARN  | qtp366157713-98  | WriteTransactionsHandler         | 230 - org.opendaylight.controller.samples.clustering-it-provider - 1.9.0.SNAPSHOT | Unable to ensure IdInts list for id: prefix-2 exists.
org.opendaylight.controller.md.sal.common.api.data.TransactionCommitFailedException: canCommit encountered an unexpected failure
        at org.opendaylight.controller.sal.core.compat.LegacyDOMDataBrokerAdapter$1.newWithCause(LegacyDOMDataBrokerAdapter.java:79) ~[?:?]
        at org.opendaylight.controller.sal.core.compat.LegacyDOMDataBrokerAdapter$1.newWithCause(LegacyDOMDataBrokerAdapter.java:68) ~[?:?]
        at org.opendaylight.yangtools.util.concurrent.ExceptionMapper.apply(ExceptionMapper.java:106) ~[287:org.opendaylight.yangtools.util:2.1.6]
        at org.opendaylight.controller.sal.core.compat.LegacyDOMDataBrokerAdapter$DOMDataTransactionAdapter$1.onFailure(LegacyDOMDataBrokerAdapter.java:303) ~[?:?]
...
Caused by: org.opendaylight.controller.cluster.access.commands.DeadTransactionException: Transactions [[0..1546]] have been purged
        at org.opendaylight.controller.cluster.datastore.AbstractFrontendHistory.handleTransactionRequest(AbstractFrontendHistory.java:88) ~[?:?]
        at org.opendaylight.controller.cluster.datastore.LeaderFrontendState.handleTransactionRequest(LeaderFrontendState.java:211) ~[?:?]
        at org.opendaylight.controller.cluster.datastore.Shard.handleRequest(Shard.java:562) ~[?:?]
        at org.opendaylight.controller.cluster.datastore.Shard.handleRequestEnvelope(Shard.java:389) ~[?:?]
        at org.opendaylight.controller.cluster.datastore.Shard.handleNonRaftCommand(Shard.java:316) ~[?:?]

OpenDaylight JIRA

[CONTROLLER-1881] DeadTransactionException failure in ELM longevity test