Uploaded image for project: 'controller'
  1. controller
  2. CONTROLLER-1684

OutOfOrderRequestException when removing shard replica under load

    XMLWordPrintable

Details

    • Bug
    • Status: Resolved
    • Resolution: Duplicate
    • None
    • None
    • clustering
    • None
    • Operating System: All
      Platform: All

    • 8486

    Description

      This is similar to CONTROLLER-1664 but the scenario is different.

      This time we have a transaction producer (module-based shard, tell-based protocol) in the same member as the shard leader, and we call remove-shard-replica.

      Robot sees OutOfOrderRequestException [0].
      Karaf.log [1] shows more. First there are many warnings:
      2017-05-17 02:49:32,274 | WARN | lt-dispatcher-31 | TransmitQueue | 197 - org.opendaylight.controller.cds-access-client - 1.1.0.Carbon | No request matching Envelope{sessionId=1, txSequence=7d0, message=ModifyTransactionSuccess{target=member-2-datastore-config-fe-0-chn-2-txn-1001-0, sequence=0}} found, ignoring response

      The error is in the middle of the warning, starting with:
      2017-05-17 02:49:32,505 | WARN | lt-dispatcher-23 | ConcurrentDOMDataBroker | 199 - org.opendaylight.controller.sal-distributed-datastore - 1.5.0.Carbon | Tx: DOM-CHAIN-1-1216 Error during phase CAN_COMMIT, starting Abort
      org.opendaylight.controller.cluster.access.commands.OutOfOrderRequestException: Expecting request 0
      at org.opendaylight.controller.cluster.datastore.AbstractFrontendHistory.<clinit>(AbstractFrontendHistory.java:47)
      at org.opendaylight.controller.cluster.datastore.FrontendHistoryMetadataBuilder.toLeaderState(FrontendHistoryMetadataBuilder.java:85)
      at org.opendaylight.controller.cluster.datastore.FrontendClientMetadataBuilder.toLeaderState(FrontendClientMetadataBuilder.java:172)
      at org.opendaylight.controller.cluster.datastore.FrontendMetadata.lambda$toLeaderState$0(FrontendMetadata.java:128)
      at com.google.common.collect.Maps$7.transformEntry(Maps.java:1812)
      at com.google.common.collect.Maps$10.getValue(Maps.java:1857)
      at java.util.HashMap.putMapEntries(HashMap.java:513)
      at java.util.HashMap.<init>(HashMap.java:489)
      at org.opendaylight.controller.cluster.datastore.FrontendMetadata.toLeaderState(FrontendMetadata.java:128)
      at org.opendaylight.controller.cluster.datastore.Shard.onLeaderChanged(Shard.java:830)
      at org.opendaylight.controller.cluster.raft.RaftActor.handleBehaviorChange(RaftActor.java:517)

      [0] https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/692/archives/log.html.gz#s1-s20-t1-k2-k9-k1
      [1] https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/692/archives/odl2_karaf.log.gz

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              Unassigned Unassigned
              vrpolak Vratko Polak
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: