https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/CONTROLLER-1628 controller <p>Latest Carbon code. <br/> 3-node cluster as part of an openstack solution.<br/> Connecting 200 computes(OVSs).<br/> Log is filled in non-stopping WARNs:<br/> WARN | lt-dispatcher-15 | FrontendClientMetadataBuilder | 214 - org.opendaylight.controller.sal-distributed-datastore - 1.5.0.SNAPSHOT | Unknown history for commited transaction member-3-datastore-operational-fe-0-chn-6-txn-24-0, ignoring</p> <p>Operating System: All<br/> Platform: All</p> CONTROLLER-1628

Unknown history for commited transaction

Bug Resolved Done Robert Varga Guy Sela Wed, 5 Apr 2017 13:39:15 +0000 Mon, 3 Jun 2019 16:45:55 +0000 Mon, 3 Jun 2019 16:45:55 +0000 Neon Fluorine SR1 Oxygen SR4 Sodium Neon SR2 clustering 0 3 <p>Attachment odl1karaf.zip has been added with description: ODL 1 log</p> <p>Attachment odl2karaf.zip has been added with description: ODL 2 log</p> <p>Attachment odl3karaf.zip has been added with description: ODL 3 log</p> <p>Guy, can you run this with debug enabled on org.opendaylight.controller.cluster.datastore?</p> <p>It looks like we have lost the record of a transaction chain being created somehow <img class="emoticon" src="https://jira.opendaylight.org/images/icons/emoticons/sad.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>(In reply to Robert Varga from comment #3)<br/> &gt; Guy, can you run this with debug enabled on<br/> &gt; org.opendaylight.controller.cluster.datastore?<br/> &gt; <br/> &gt; It looks like we have lost the record of a transaction chain being created<br/> &gt; somehow <img class="emoticon" src="https://jira.opendaylight.org/images/icons/emoticons/sad.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>Sure, anything to take that upset emoji off your face <img class="emoticon" src="https://jira.opendaylight.org/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/><br/> I'll do it tomorrow.</p> <p>Attachment odl3.tar.gz has been added with description: ODL3 LOG with debug</p> <p>Attachment odl2.tar.gz has been added with description: ODL2 LOG with debug</p> <p>Attachment odl1.tar.gz has been added with description: ODL1 LOG with debug</p> <p>So this is a follower-side of things, but unfortunately the logs are incomplete. The following histories are candidates:</p> <p>member-3-datastore-operational-fe-0-chn-1<br/> member-3-datastore-operational-fe-0-chn-3<br/> member-3-datastore-operational-fe-0-chn-6<br/> member-3-datastore-operational-fe-0-chn-9<br/> member-3-datastore-operational-fe-0-chn-11<br/> member-3-datastore-operational-fe-0-chn-12</p> <p>But the logs from master (odl3) do not contain the records of those chains starting nor is there overlap between transactions on master and follower.</p> <p>Digging into journal entries, the most recent inconsistency is detected at:</p> <p>2017-04-09 17:20:40,052 | DEBUG | lt-dispatcher-34 | Shard | 207 - org.opendaylight.controller.sal-clustering-commons - 1.5.0.SNAPSHOT | member-2-shard-topology-operational: Applying state for log index 817 data org.opendaylight.controller.cluster.datastore.persisted.CommitTransactionPayload@54aae041</p> <p>but the logs are missing that part, because the oldest we have on the leader side is:</p> <p>2017-04-09 17:20:45,030 | DEBUG | lt-dispatcher-40 | Shard | 207 - org.opendaylight.controller.sal-clustering-commons - 1.5.0.SNAPSHOT | member-3-shard-topology-operational (Leader): handleAppendEntriesReply - FollowerLogInformation for member-2-shard-topology-operational updated: matchIndex: 846, nextIndex: 847</p> <p>At the very least we need some overlap in the logs related to the what state was exchanged between leader and follower. The best would be to have logs containing the complete transaction chain lifetime (i.e. from its first transaction).</p> <p>patch to improve logging details: <a href="https://git.opendaylight.org/gerrit/54884" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/54884</a></p> <p>Hi Guy,<br/> can you review the last comment from Robert and attach new logs for us?<br/> Thank you.</p> <p>We were on holiday until now. I will try to produce the logs today.</p> <p>Attachment odl1.tar.gz has been added with description: ODL 1 Log with Patch</p> <p>Attachment odl2.tar.gz has been added with description: ODL 2 Log with Patch</p> <p>Attachment odl3.tar.gz has been added with description: ODL 3 Log with Patch</p> <p>Thans Guy, but the logs are still incomplete &#8211; they seem to be only last 3 log files (.10, .9, .8) for each node. I really do need the complete thing, as the local history is communicated when a transaction chain is first created and on snapshot transfers. None of that is captured in the files <img class="emoticon" src="https://jira.opendaylight.org/images/icons/emoticons/sad.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>Attachment odl1full.tar.gz has been added with description: ODL1 Full Log</p> <p>Attachment odl2A.tar.gz has been added with description: ODL2 Part A Full Log</p> <p>Attachment odl2B.tar.gz has been added with description: ODL2 Part B Full Log</p> <p>Attachment odl3A.tar.gz has been added with description: ODL3 Part A Full Log</p> <p>Attachment odl3B.tar.gz has been added with description: ODL3 Part B Full Log</p> <p>Attachment odl3C.tar.gz has been added with description: ODL3 Part C Full Log</p> <p>Done</p> <p>Thanks, this looks promising <img class="emoticon" src="https://jira.opendaylight.org/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>The problem is that ShardCommitCoordinator.handleReadyLocalTransaction() invokes ShardDataTree.createReadyCohort() directly rather than going through transaction chain tracking in ShardDataTree.</p> <p>This is a consistency problem in ShardDataTree interfaces, as ShardDataTree.newReadWriteTransaction() actually does the tracking, leaving ShardCommitCoordinator totally ignorant of it.</p> <p>carbon: <a href="https://git.opendaylight.org/gerrit/55739" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/55739</a></p> <p>Agreed on killing DataTreeCohortActor. I have filed <a href="https://jira.opendaylight.org/browse/CONTROLLER-1641" title="Integrate DataTreeCohortActor" class="issue-link" data-issue-key="CONTROLLER-1641"><del>CONTROLLER-1641</del></a> to track that.</p> <p>Ugh, that comment should have gone someplace else.</p> <p>This is still occurring in testing, notably in <a href="https://jira.opendaylight.org/browse/CONTROLLER-1664" title="C: OutOfOrderRequestException: Expecting request 0 (while create-prefix-shard)" class="issue-link" data-issue-key="CONTROLLER-1664"><del>CONTROLLER-1664</del></a>. It happens on clean leader movement, when the leader transitions to being a follower.</p> <p>Logs indicate that the history was known when in leader role:</p> <p>2017-05-15 12:48:06,725 | DEBUG | lt-dispatcher-22 | AbstractFrontendHistory | 199 - org.opendaylight.controller.sal-distributed-datastore - 1.5.0.SNAPSHOT | member-2-shard-default-config: allocating new open transaction member-3-datastore-config-fe-0-chn-2-txn-0-0</p> <p>but after transition to follower:</p> <p>2017-05-15 12:48:06,828 | INFO | lt-dispatcher-22 | Shard | 192 - org.opendaylight.controller.sal-clustering-commons - 1.5.0.SNAPSHOT | member-2-shard-default-config (Leader) :- Switching from behavior Leader to Follower, election term: 3</p> <p>we get:</p> <p>2017-05-15 12:49:11,865 | WARN | lt-dispatcher-26 | FrontendClientMetadataBuilder | 199 - org.opendaylight.controller.sal-distributed-datastore - 1.5.0.SNAPSHOT | member-2-shard-default-config: Unknown history for commited transaction member-3-datastore-config-fe-0-chn-2-txn-2-0, ignoring</p> <p>The transaction chain in question is a long-lived one, which survives the leader ship change event.</p> <p>This evidence points to towards leader-side persistence codepath, which does not seem to apply lifecycle events to its own FrontendClientMetadataBuilder.</p> <p><a href="https://git.opendaylight.org/gerrit/57138" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/57138</a></p> <p>This is still surfacing in Genius 3-node CSIT.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>2018-11-26T08:54:04,046 | DEBUG | opendaylight-cluster-data-shard-dispatcher-20 | FrontendClientMetadataBuilder    | 232 - org.opendaylight.controller.sal-distributed-datastore - 1.9.0.SNAPSHOT | member-3-shard-default-operational: Created local history LocalHistoryIdentifier{client=ClientIdentifier{frontend=member-2-frontend-datastore-operational, generation=0}, history=4, cookie=0} 2018-11-26T08:54:04,047 | DEBUG | opendaylight-cluster-data-shard-dispatcher-20 | FrontendClientMetadataBuilder    | 232 - org.opendaylight.controller.sal-distributed-datastore - 1.9.0.SNAPSHOT | member-3-shard-default-operational: Purged transaction member-2-datastore-operational-fe-0-txn-177-0 2018-11-26T08:54:04,050 | DEBUG | opendaylight-cluster-data-shard-dispatcher-44 | FrontendClientMetadataBuilder    | 232 - org.opendaylight.controller.sal-distributed-datastore - 1.9.0.SNAPSHOT | member-3-shard-default-operational: Created local history LocalHistoryIdentifier{client=ClientIdentifier{frontend=member-2-frontend-datastore-operational, generation=0}, history=5, cookie=0} 2018-11-26T08:54:04,050 | DEBUG | opendaylight-cluster-data-shard-dispatcher-44 | FrontendClientMetadataBuilder    | 232 - org.opendaylight.controller.sal-distributed-datastore - 1.9.0.SNAPSHOT | member-3-shard-default-operational: Closed history LocalHistoryIdentifier{client=ClientIdentifier{frontend=member-2-frontend-datastore-operational, generation=0}, history=4, cookie=0} 2018-11-26T08:54:04,051 | DEBUG | opendaylight-cluster-data-shard-dispatcher-44 | FrontendClientMetadataBuilder    | 232 - org.opendaylight.controller.sal-distributed-datastore - 1.9.0.SNAPSHOT | member-3-shard-default-operational: Purged history LocalHistoryIdentifier{client=ClientIdentifier{frontend=member-2-frontend-datastore-operational, generation=0}, history=4, cookie=0} 2018-11-26T08:54:04,051 | WARN  | opendaylight-cluster-data-shard-dispatcher-44 | FrontendClientMetadataBuilder    | 232 - org.opendaylight.controller.sal-distributed-datastore - 1.9.0.SNAPSHOT | member-3-shard-default-operational: Unknown history for commited transaction member-2-datastore-operational-fe-0-chn-4-txn-0-0, ignoring 2018-11-26T08:54:04,054 | DEBUG | opendaylight-cluster-data-shard-dispatcher-42 | FrontendClientMetadataBuilder    | 232 - org.opendaylight.controller.sal-distributed-datastore - 1.9.0.SNAPSHOT | member-3-shard-topology-operational: Committed transaction member-1-datastore-operational-fe-0-chn-1-txn-54-0 2018-11-26T08:54:04,058 | WARN  | opendaylight-cluster-data-shard-dispatcher-45 | FrontendClientMetadataBuilder    | 232 - org.opendaylight.controller.sal-distributed-datastore - 1.9.0.SNAPSHOT | member-3-shard-default-operational: Unknown history for purged transaction member-2-datastore-operational-fe-0-chn-4-txn-0-0, ignoring 2018-11-26T08:54:04,060 | DEBUG | opendaylight-cluster-data-shard-dispatcher-42 | FrontendClientMetadataBuilder    | 232 - org.opendaylight.controller.sal-distributed-datastore - 1.9.0.SNAPSHOT | member-3-shard-topology-operational: Purged transaction member-1-datastore-operational-fe-0-chn-1-txn-54-0 2018-11-26T08:54:04,068 | DEBUG | opendaylight-cluster-data-shard-dispatcher-42 | FrontendClientMetadataBuilder    | 232 - org.opendaylight.controller.sal-distributed-datastore - 1.9.0.SNAPSHOT | member-3-shard-default-operational: Committed transaction member-1-datastore-operational-fe-0-txn-115-0 2018-11-26T08:54:04,081 | DEBUG | opendaylight-cluster-data-shard-dispatcher-43 | FrontendClientMetadataBuilder    | 232 - org.opendaylight.controller.sal-distributed-datastore - 1.9.0.SNAPSHOT | member-3-shard-default-operational: Closed history LocalHistoryIdentifier{client=ClientIdentifier{frontend=member-2-frontend-datastore-operational, generation=0}, history=5, cookie=0} 2018-11-26T08:54:04,081 | DEBUG | opendaylight-cluster-data-shard-dispatcher-43 | FrontendClientMetadataBuilder    | 232 - org.opendaylight.controller.sal-distributed-datastore - 1.9.0.SNAPSHOT | member-3-shard-default-operational: Purged history LocalHistoryIdentifier{client=ClientIdentifier{frontend=member-2-frontend-datastore-operational, generation=0}, history=5, cookie=0} 2018-11-26T08:54:04,082 | DEBUG | opendaylight-cluster-data-shard-dispatcher-43 | FrontendClientMetadataBuilder    | 232 - org.opendaylight.controller.sal-distributed-datastore - 1.9.0.SNAPSHOT | member-3-shard-default-operational: Purged transaction member-1-datastore-operational-fe-0-txn-115-0 2018-11-26T08:54:04,082 | WARN  | opendaylight-cluster-data-shard-dispatcher-43 | FrontendClientMetadataBuilder    | 232 - org.opendaylight.controller.sal-distributed-datastore - 1.9.0.SNAPSHOT | member-3-shard-default-operational: Unknown history for commited transaction member-2-datastore-operational-fe-0-chn-5-txn-0-0, ignoring 2018-11-26T08:54:04,086 | DEBUG | opendaylight-cluster-data-shard-dispatcher-42 | FrontendClientMetadataBuilder    | 232 - org.opendaylight.controller.sal-distributed-datastore - 1.9.0.SNAPSHOT | member-3-shard-topology-operational: Committed transaction member-1-datastore-operational-fe-0-chn-1-txn-55-0 2018-11-26T08:54:04,089 | DEBUG | opendaylight-cluster-data-shard-dispatcher-42 | FrontendClientMetadataBuilder    | 232 - org.opendaylight.controller.sal-distributed-datastore - 1.9.0.SNAPSHOT | member-3-shard-topology-operational: Purged transaction member-1-datastore-operational-fe-0-chn-1-txn-55-0 2018-11-26T08:54:04,097 | DEBUG | opendaylight-cluster-data-shard-dispatcher-45 | FrontendClientMetadataBuilder    | 232 - org.opendaylight.controller.sal-distributed-datastore - 1.9.0.SNAPSHOT | member-3-shard-default-config: Aborted transaction member-2-datastore-config-fe-0-txn-214-0 2018-11-26T08:54:04,098 | DEBUG | opendaylight-cluster-data-shard-dispatcher-45 | FrontendClientMetadataBuilder    | 232 - org.opendaylight.controller.sal-distributed-datastore - 1.9.0.SNAPSHOT | member-3-shard-default-config: Committed transaction member-2-datastore-config-fe-0-txn-216-0 2018-11-26T08:54:04,098 | DEBUG | opendaylight-cluster-data-shard-dispatcher-45 | FrontendClientMetadataBuilder    | 232 - org.opendaylight.controller.sal-distributed-datastore - 1.9.0.SNAPSHOT | member-3-shard-default-config: Aborted transaction member-2-datastore-config-fe-0-txn-215-0 2018-11-26T08:54:04,105 | DEBUG | opendaylight-cluster-data-shard-dispatcher-45 | FrontendClientMetadataBuilder    | 232 - org.opendaylight.controller.sal-distributed-datastore - 1.9.0.SNAPSHOT | member-3-shard-default-config: Aborted transaction member-2-datastore-config-fe-0-txn-217-0 2018-11-26T08:54:04,105 | DEBUG | opendaylight-cluster-data-shard-dispatcher-45 | FrontendClientMetadataBuilder    | 232 - org.opendaylight.controller.sal-distributed-datastore - 1.9.0.SNAPSHOT | member-3-shard-default-config: Aborted transaction member-2-datastore-config-fe-0-txn-218-0 2018-11-26T08:54:04,106 | WARN  | opendaylight-cluster-data-shard-dispatcher-42 | FrontendClientMetadataBuilder    | 232 - org.opendaylight.controller.sal-distributed-datastore - 1.9.0.SNAPSHOT | member-3-shard-default-operational: Unknown history for purged transaction member-2-datastore-operational-fe-0-chn-5-txn-0-0, ignoring </pre> </div></div> <p>This points towards CloseTransactionChain being too eager with purge. We are still tracking an unpurged transaction in the chain, hence we should not be removing it. The above happens if the user does:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> TransactionChain chain; WriteTransaction tx = chain.newWriteOnlyTransaction(); <span class="code-comment">// some modifications </span>Future&lt;?&gt; <span class="code-keyword">future</span> = tx.commit(); chain.close(); </pre> </div></div> <p>Transaction commit will be waiting in the queue, while chain.close() will have an immediate effect – hence when the transaction completes, FrontendClientMetadataBuilder already has no record of the history.</p> <p>Under normal operation this has no ill effects aside from the annoying warning. During failover operations, this may cause the transaction to be aborted, as the new leader will have no memory of the history, preventing the transaction from being re-created.</p> <p>tell-based protocol is unaffected as far as I can tell, as it is using two separate messages {Destroy,Purge}HistoryRequest instead of a single CloseTransactionChain.</p> <p>This will require postponing the purge until after all transactions are done. Tracking them down, though, may be a bit tricky, as they different states are tracked in different places (pending, committing, committed, purged). Since this is only for ask-based, we can probably not care about committed (a.k.a. closed) transactions and need to track down only transactions which are being committed.</p> Blocks CONTROLLER-1653 Duplicate CONTROLLER-1638 Development External issue ID 8159 External issue URL Issue Type ODL SR Target Milestone Rank 0|i02rqv: