[CONTROLLER-1684] OutOfOrderRequestException when removing shard replica under load

Duplicate — Tue, 25 Jul 2023 08:24:38 +0000

This is similar to ~~CONTROLLER-1664~~ but the scenario is different.

This time we have a transaction producer (module-based shard, tell-based protocol) in the same member as the shard leader, and we call remove-shard-replica.

Robot sees OutOfOrderRequestException [0].
Karaf.log [1] shows more. First there are many warnings:
2017-05-17 02:49:32,274 | WARN | lt-dispatcher-31 | TransmitQueue | 197 - org.opendaylight.controller.cds-access-client - 1.1.0.Carbon | No request matching Envelope{sessionId=1, txSequence=7d0, message=ModifyTransactionSuccess{target=member-2-datastore-config-fe-0-chn-2-txn-1001-0, sequence=0}} found, ignoring response

The error is in the middle of the warning, starting with:
2017-05-17 02:49:32,505 | WARN | lt-dispatcher-23 | ConcurrentDOMDataBroker | 199 - org.opendaylight.controller.sal-distributed-datastore - 1.5.0.Carbon | Tx: DOM-CHAIN-1-1216 Error during phase CAN_COMMIT, starting Abort
org.opendaylight.controller.cluster.access.commands.OutOfOrderRequestException: Expecting request 0
at org.opendaylight.controller.cluster.datastore.AbstractFrontendHistory.<clinit>(AbstractFrontendHistory.java:47)
at org.opendaylight.controller.cluster.datastore.FrontendHistoryMetadataBuilder.toLeaderState(FrontendHistoryMetadataBuilder.java:85)
at org.opendaylight.controller.cluster.datastore.FrontendClientMetadataBuilder.toLeaderState(FrontendClientMetadataBuilder.java:172)
at org.opendaylight.controller.cluster.datastore.FrontendMetadata.lambda$toLeaderState$0(FrontendMetadata.java:128)
at com.google.common.collect.Maps$7.transformEntry(Maps.java:1812)
at com.google.common.collect.Maps$10.getValue(Maps.java:1857)
at java.util.HashMap.putMapEntries(HashMap.java:513)
at java.util.HashMap.<init>(HashMap.java:489)
at org.opendaylight.controller.cluster.datastore.FrontendMetadata.toLeaderState(FrontendMetadata.java:128)
at org.opendaylight.controller.cluster.datastore.Shard.onLeaderChanged(Shard.java:830)
at org.opendaylight.controller.cluster.raft.RaftActor.handleBehaviorChange(RaftActor.java:517)

[0] https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/692/archives/log.html.gz#s1-s20-t1-k2-k9-k1
[1] https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/692/archives/odl2_karaf.log.gz

OpenDaylight JIRA

[CONTROLLER-1684] OutOfOrderRequestException when removing shard replica under load