https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/CONTROLLER-1687 controller <p>This happened once so far, usually the failure happens earlier (<a href="https://jira.opendaylight.org/browse/CONTROLLER-1679" title="DataStoreUnavailableException sooner than 60 seconds after isolation" class="issue-link" data-issue-key="CONTROLLER-1679"><del>CONTROLLER-1679</del></a>).<br/> This behavior is expected to change after the fix <span class="error">&#91;0&#93;</span> goes in, we will probably reuse this Bug if the failure does not happen after that.</p> <p>The suite calls write-transactions on a member, then isolates the member, then waits some time (60 second right now) expecting no failure, and then some time (20 s) expecting a failure.</p> <p>This failure (or any other response) did not happen <span class="error">&#91;1&#93;</span> when the target module-based shard has leader on the same member. Tell-based protocol, simple (as in not chained) transactions. In kkaraf.log <span class="error">&#91;2&#93;</span> between 13:46:27,686 and 13:50:38,617 I see usual isolation-related messages only.</p> <p><span class="error">&#91;0&#93;</span> <a href="https://git.opendaylight.org/gerrit/57074" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/57074</a><br/> <span class="error">&#91;1&#93;</span> <a href="https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/693/archives/log.html.gz#s1-s32-t3-k2-k17-k1-k1" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/693/archives/log.html.gz#s1-s32-t3-k2-k17-k1-k1</a><br/> <span class="error">&#91;2&#93;</span> <a href="https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/693/archives/odl2_karaf.log.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/693/archives/odl2_karaf.log.gz</a></p> <p>Operating System: All<br/> Platform: All</p> CONTROLLER-1687

Transaction writer does not fail expected time after isolation

Bug Resolved Done Robert Varga Vratko Polak Wed, 17 May 2017 17:17:15 +0000 Tue, 25 Jul 2023 08:24:39 +0000 Thu, 27 Jul 2017 09:47:02 +0000 clustering 0 4 <p>I think with tell based protocol the correct time to wait until the transaction should time out is 2 minutes after: <a href="https://git.opendaylight.org/gerrit/#/c/56874/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/56874/</a></p> <p>Currently, the suite waits for 121 seconds <span class="error">&#91;0&#93;</span>, which may be too close. The strange thing is that the responses report different times <span class="error">&#91;1&#93;</span> timeout happened. The smallest time is 128 seconds, other times are 162s and 174s.</p> <p>Should we make the suite more tolerant to the timeout timing?</p> <p><span class="error">&#91;0&#93;</span> <a href="https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/715/archives/log.html.gz#s1-s28-t3-k2-k2" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/715/archives/log.html.gz#s1-s28-t3-k2-k2</a><br/> <span class="error">&#91;1&#93;</span> <a href="https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/715/archives/log.html.gz#s1-s28-t3-k2-k26-k1-k7" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/715/archives/log.html.gz#s1-s28-t3-k2-k26-k1-k7</a></p> <p>Something similar happened in another scenario. Still module-based shard with tell-based protocol, but now only one writer (co-located with the leader).</p> <p>The suite computes end time from the time transactions stream started <span class="error">&#91;2&#93;</span>, which is less precise than computing from isolation time. But the difference is less than 6 seconds, and we wait for the failure for additional 20 seconds, so I believe it is the same Bug.</p> <p><span class="error">&#91;2&#93;</span> <a href="https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/715/archives/log.html.gz#s1-s32-t3-k2-k12" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/715/archives/log.html.gz#s1-s32-t3-k2-k12</a></p> <p>&gt; module-based shard</p> <p>Prefix-based shards now behave the same way <span class="error">&#91;3&#93;</span>.</p> <p>The interesting fact is that in the last run, this Bug always happened when writer and leader were on the same member, but never when the writer was isolated on a follower member.</p> <p><span class="error">&#91;3&#93;</span> <a href="https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/715/archives/log.html.gz#s1-s34-t1-k2-k19-k1-k1" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/715/archives/log.html.gz#s1-s34-t1-k2-k19-k1-k1</a></p> <p>seems to be similar problem as #8445. We need to have debug logs, to confirm a duplicat.</p> <p>New run with debug logs <a href="https://jenkins.opendaylight.org/sandbox/job/controller-csit-3node-clustering-only-carbon-4th/5/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/sandbox/job/controller-csit-3node-clustering-only-carbon-4th/5/</a></p> <p>Odl bundle used from <a href="https://jenkins.opendaylight.org/releng/view/integration/job/integration-multipatch-test-carbon/37/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/integration/job/integration-multipatch-test-carbon/37/</a></p> <p>Run with debug logs (includung akka)<br/> <a href="https://jenkins.opendaylight.org/sandbox/job/controller-csit-3node-clustering-only-carbon-4th/6/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/sandbox/job/controller-csit-3node-clustering-only-carbon-4th/6/</a></p> <p>Odl build from from controller=63/57763/2:99/57699/3<br/> <a href="https://nexus.opendaylight.org/content/repositories/opendaylight.snapshot/org/opendaylight/integration/integration/distribution/distribution-karaf/0.6.0-SNAPSHOT/distribution-karaf-0.6.0-20170524.144642-162.zip" class="external-link" target="_blank" rel="nofollow noopener">https://nexus.opendaylight.org/content/repositories/opendaylight.snapshot/org/opendaylight/integration/integration/distribution/distribution-karaf/0.6.0-SNAPSHOT/distribution-karaf-0.6.0-20170524.144642-162.zip</a></p> <p>New logs (akka included)<br/> <a href="https://jenkins.opendaylight.org/sandbox/job/controller-csit-3node-clustering-only-carbon-4th/7/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/sandbox/job/controller-csit-3node-clustering-only-carbon-4th/7/</a></p> <p>Odl built from controller=70/57770/4:99/57699/3 at<br/> <a href="https://jenkins.opendaylight.org/releng/view/integration/job/integration-multipatch-test-carbon/45/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/integration/job/integration-multipatch-test-carbon/45/</a></p> <p>Not-failed rpc:<br/> <a href="https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-only-carbon-4th/7/archives/log.html.gz#s1-s2-t1-k2-k20-k1-k2-k1" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-only-carbon-4th/7/archives/log.html.gz#s1-s2-t1-k2-k20-k1-k2-k1</a></p> <p>With <span class="error">&#91;11&#93;</span> codebase this sometimes happens (writer does not fail within 30 seconds) <span class="error">&#91;12&#93;</span> and sometimes not (writer fails as expected, in just three seconds) <span class="error">&#91;13&#93;</span>.</p> <p><span class="error">&#91;11&#93;</span> <a href="https://git.opendaylight.org/gerrit/#/c/57822/1" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/57822/1</a><br/> <span class="error">&#91;12&#93;</span> <a href="https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-only-carbon/4/archives/log.html.gz#s1-s14-t1-k2-k19-k1-k1" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-only-carbon/4/archives/log.html.gz#s1-s14-t1-k2-k19-k1-k1</a><br/> <span class="error">&#91;13&#93;</span> <a href="https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-only-carbon/4/archives/log.html.gz#s1-s14-t3-k2-k19" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-only-carbon/4/archives/log.html.gz#s1-s14-t3-k2-k19</a></p> <p>In those particular runs I do not see anything unexpected. It would seem our timer is not running in ReconnectingConnection (the original timer is stopped when it reaches reconnection).</p> <p>I have noticed the following in what looks to be a user thead:</p> <p>2017-05-29 03:13:03,006 | DEBUG | pool-30-thread-1 | AbstractClientConnection | 197 - org.opendaylight.controller.cds-access-client - 1.1.0.SNAPSHOT | Sleeping for 2654453ms</p> <p>Which means that thread is being blocked for almost 45 minutes, and that came form the ProgressTracker. The is the first 'Sleeping for' message, which is indicating we were not exerting backpressure while the connection was connecting. That process finished just a millisecond before:</p> <p>2017-05-29 03:13:03,005 | INFO | ult-dispatcher-4 | ClientActorBehavior | 197 - org.opendaylight.controller.cds-access-client - 1.1.0.SNAPSHOT | member-1-frontend-datastore-config: replaced connection ReconnectingClientConnection{client=ClientIdentifier</p> {frontend=member-1-frontend-datastore-config, generation=0}, cookie=0, backend=ShardBackendInfo{actor=Actor<a href="#-1620047755" target="_blank" rel="noopener">akka://opendaylight-cluster-data/user/shardmanager-config/member-1-shard-default-config#-1620047755</a>, sessionId=0, version=BORON, maxMessages=1000, cookie=0, shard=default, dataTree present=true}} with ConnectedClientConnection{client=ClientIdentifier{frontend=member-1-frontend-datastore-config, generation=0} <p>, cookie=0, backend=ShardBackendInfo{actor=Actor<span class="error">&#91;akka.tcp://opendaylight-cluster-data@10.29.15.15:2550/user/shardmanager-config/member-2-shard-default-config#-20940815&#93;</span>, sessionId=1, version=BORON, maxMessages=1000, cookie=0, shard=default, dataTree present=false}}</p> <p>We clearly need to put an upper bound on sleeping, but also we need fix the backpressure &#8211; the client should have been throttled before that point (i.e. there should have been 'Sleeping for' messages.</p> <p>Note this is the 'reconnecting' case, the connection has previously been connected and there is a forwarder involved.</p> <p><a href="https://git.opendaylight.org/gerrit/57996" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/57996</a></p> <p>(In reply to Robert Varga from comment #13)<br/> &gt; <a href="https://git.opendaylight.org/gerrit/57996" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/57996</a></p> <p>The problem still present.</p> <p>Job: <a href="https://jenkins.opendaylight.org/sandbox/job/controller-csit-3node-clustering-only-carbon-4th/2/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/sandbox/job/controller-csit-3node-clustering-only-carbon-4th/2/</a><br/> Problem: <a href="https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-only-carbon-4th/2/archives/log.html.gz#s1-s2-t1-k2-k17-k1-k2-k1" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-only-carbon-4th/2/archives/log.html.gz#s1-s2-t1-k2-k17-k1-k2-k1</a><br/> Zip built from: controller=96/57996/5:99/57699/5</p> <p><a href="https://git.opendaylight.org/gerrit/58194" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/58194</a></p> <p>&gt; <a href="https://git.opendaylight.org/gerrit/58194" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/58194</a></p> <p>Build that had 58194/3 chain with 57699/5 on top still failed <span class="error">&#91;18&#93;</span> on Sandbox.<br/> That run was without debug messages, so karaf.log <span class="error">&#91;19&#93;</span> does not show much between 07:29:35,869 and 07:32:13,384.</p> <p><span class="error">&#91;18&#93;</span> <a href="https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-only-carbon/4/log.html.gz#s1-s16-t1-k2-k16-k1-k1" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-only-carbon/4/log.html.gz#s1-s16-t1-k2-k16-k1-k1</a><br/> <span class="error">&#91;19&#93;</span> <a href="https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-only-carbon/4/odl1_karaf.log.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-only-carbon/4/odl1_karaf.log.gz</a></p> <p>&gt; Build that had 58194/3 chain with 57699/5 on top</p> <p>The same run with some CDS related debug messages. The karaf.log <span class="error">&#91;20&#93;</span> of the member with transaction producer and original prefix shard leader is huge.<br/> Transaction producer starts at 18:49:52,337, impeding isolation is announced at 18:50:02,850, progress is stopped at 18:50:03,842, rejoin (after the test failed) is announced at 18:52:23,138.</p> <p><span class="error">&#91;20&#93;</span> <a href="https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-only-carbon/6/odl3_karaf.log.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-only-carbon/6/odl3_karaf.log.gz</a></p> <p>The victim transaction should be:</p> <p>2017-06-06 18:51:56,112 | WARN | lt-dispatcher-20 | ShardDataTree | 199 - org.opendaylight.controller.sal-distributed-datastore - 1.5.1.SNAPSHOT | member-3-shard-id-ints!-config: Current transaction member-3-datastore-Shard-id-ints!-fe-0-chn-5-txn-2853-0 has timed out after 15000 ms in state COMM<br/> IT_PENDING<br/> 2017-06-06 18:51:56,112 | WARN | lt-dispatcher-20 | ShardDataTree | 199 - org.opendaylight.controller.sal-distributed-datastore - 1.5.1.SNAPSHOT | member-3-shard-id-ints!-config: Transaction member-3-datastore-Shard-id-ints!-fe-0-chn-5-txn-2853-0 is still committing, cannot abort</p> <p>It's request happened here:</p> <p>2017-06-06 18:50:03,702 | DEBUG | lt-dispatcher-28 | AbstractProxyTransaction | 199 - org.opendaylight.controller.sal-distributed-datastore - 1.5.1.SNAPSHOT | Transaction proxy LocalReadWriteProxyTransaction</p> {identifier=member-3-datastore-Shard-id-ints!-fe-0-chn-5-txn-2853-0, state=FLUSHED} sending request Co<br/> mmitLocalTransactionRequest{target=member-3-datastore-Shard-id-ints!-fe-0-chn-5-txn-2853-0, sequence=0, replyTo=Actor[akka://opendaylight-cluster-data/user/$m#531635654], coordinated=false} callback org.opendaylight.controller.cluster.databroker.actors.dds.AbstractProxyTransaction$$Lambda$128/1535548548@1a032e87<br/> <br/> <br/> and the timeout occurred pretty much spot-on:<br/> <br/> 2017-06-06 18:52:03,732 | DEBUG | lt-dispatcher-27 | AbstractClientConnection | 197 - org.opendaylight.controller.cds-access-client - 1.1.1.SNAPSHOT | member-3-frontend-datastore-Shard-id-ints!: running timer on ReconnectingClientConnection{client=ClientIdentifier{frontend=member-3-frontend-datastore-Shard-i d-ints!, generation=0}, cookie=0, backend=ShardBackendInfo{actor=Actor<a href="#821446964" target="_blank" rel="noopener">akka://opendaylight-cluster-data/user/shardmanager-config/member-3-shard-id-ints!-config#821446964</a>, sessionId=0, version=BORON, maxMessages=1000, cookie=0, shard=id-ints!, dataTree present=true}}<br/> 2017-06-06 18:52:03,732 | DEBUG | lt-dispatcher-27 | AbstractClientConnection | 197 - org.opendaylight.controller.cds-access-client - 1.1.1.SNAPSHOT | member-3-frontend-datastore-Shard-id-ints!: Connection ReconnectingClientConnection{client=ClientIdentifier{frontend=member-3-frontend-datastore-Shard-id-ints!, generation=0}, cookie=0, backend=ShardBackendInfo{actor=Actor<a href="#821446964" target="_blank" rel="noopener">akka://opendaylight-cluster-data/user/shardmanager-config/member-3-shard-id-ints!-config#821446964</a>, sessionId=0, version=BORON, maxMessages=1000, cookie=0, shard=id-ints!, dataTree present=true}} timed out entry TransmittedConnectionEntry{request=CommitLocalTransactionRequest{target=member-3-datastore-Shard-id-ints!-fe-0-chn-5-txn-2853-0, sequence=0, replyTo=Actor[akka://opendaylight-cluster-data/user/$m#531635654], coordinated=false}, enqueuedTicks=5430269723299, sessionId=0, txSequence=5706}<br/> <br/> and it was routed to the Future:<br/> <br/> 2017-06-06 18:52:03,815 | DEBUG | lt-dispatcher-27 | AbstractProxyTransaction | 199 - org.opendaylight.controller.sal-distributed-datastore - 1.5.1.SNAPSHOT | Transaction LocalReadWriteProxyTransaction{identifier=member-3-datastore-Shard-id-ints!-fe-0-chn-5-txn-2853-0, state=FLUSHED} <p> directCommit completed</p> <p>This is recently happening very reliably, in every test where the client is isolated for a longer time, except for one combination. That combination is when transaction writer (module-based shard) uses transaction chain (as opposed to free-standing transactions). That combination is passing last 7 days.</p> <p>It might suggest that the core works correctly, but the test driver does not detect the first transaction failure correctly.<br/> The test drivers are ProduceTransactionsHandler and WriteTransactionsHandler (which has a separate codepath for failing the RPC call when transaction chain fails).</p> <p>Sandbox run with multiple test cases relevant to this Bug, verbose logs and 59527/1 codebase: <span class="error">&#91;21&#93;</span></p> <p><span class="error">&#91;21&#93;</span> <a href="https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-ci-only-carbon/5/" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-ci-only-carbon/5/</a></p> <p><a href="https://git.opendaylight.org/gerrit/59576" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/59576</a></p> <p>Latest Sandbox run, one test case (waiting time increased to 30 minutes) against codebase <span class="error">&#91;22&#93;</span> (with cherry-pick <span class="error">&#91;23&#93;</span>) with verbose logs, still shows <span class="error">&#91;24&#93;</span> that transaction-writer using simple transactions (not chains) does not report failure when the node (member-2, containing both shard leader and writer) is isolated indefinitely.</p> <p><span class="error">&#91;22&#93;</span> <a href="https://git.opendaylight.org/gerrit/#/c/59604/1" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/59604/1</a><br/> <span class="error">&#91;23&#93;</span> <a href="https://git.opendaylight.org/gerrit/#/c/59575/1" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/59575/1</a><br/> <span class="error">&#91;24&#93;</span> <a href="https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-ci-only-carbon/11/log.html.gz#s1-s2-t1-k2-k16-k1" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-ci-only-carbon/11/log.html.gz#s1-s2-t1-k2-k16-k1</a></p> <p>&gt; one test case</p> <p>The same test case is still failing <span class="error">&#91;25&#93;</span> on releng, but other 6 test cases have passed on releng today.</p> <p><span class="error">&#91;25&#93;</span> <a href="https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/765/log.html.gz#s1-s32-t3-k2-k16-k1-k1" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/765/log.html.gz#s1-s32-t3-k2-k16-k1-k1</a></p> <p>&gt; other 6 test cases have passed</p> <p>Today only 5 of them.<br/> The another failing case is <span class="error">&#91;26&#93;</span>.</p> <p><span class="error">&#91;26&#93;</span> <a href="https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/767/log.html.gz#s1-s32-t7-k2-k16-k1-k1" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/767/log.html.gz#s1-s32-t7-k2-k16-k1-k1</a></p> <p>Still present, as two test case failing.<br/> Sandbox links to Robot expecting but not seeing writer response: <span class="error">&#91;27&#93;</span> <span class="error">&#91;28&#93;</span>.</p> <p><span class="error">&#91;27&#93;</span> <a href="https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-ci-only-carbon/1/log.html.gz#s1-s2-t3-k2-k16-k1-k1" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-ci-only-carbon/1/log.html.gz#s1-s2-t3-k2-k16-k1-k1</a><br/> <span class="error">&#91;28&#93;</span> <a href="https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-ci-only-carbon/1/log.html.gz#s1-s2-t7-k2-k16-k1" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-ci-only-carbon/1/log.html.gz#s1-s2-t7-k2-k16-k1</a></p> <p>With <span class="error">&#91;29&#93;</span> the message <span class="error">&#91;30&#93;</span> is different:<br/> "Backend timeout in state READY after 15000ms" is returned 35 second after the leader movement.</p> <p><span class="error">&#91;29&#93;</span> <a href="https://git.opendaylight.org/gerrit/#/c/59895/1" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/59895/1</a><br/> <span class="error">&#91;30&#93;</span> <a href="https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-ddb-expl-lead-movement-longevity-only-carbon/5/log.html.gz#s1-s2-t1-k2-k1-k1-k1-k1-k1-k1-k2-k1-k1-k2-k10" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-ddb-expl-lead-movement-longevity-only-carbon/5/log.html.gz#s1-s2-t1-k2-k1-k1-k1-k1-k1-k1-k2-k1-k1-k2-k10</a></p> <p>This week Sandbox run: <span class="error">&#91;31&#93;</span>.</p> <p><span class="error">&#91;31&#93;</span> <a href="https://jenkins.opendaylight.org/sandbox/job/controller-csit-3node-clustering-ci-only-carbon/2/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/sandbox/job/controller-csit-3node-clustering-ci-only-carbon/2/</a></p> <p>I think I have a fix <span class="error">&#91;32&#93;</span>, but it probably needs a rebase and a better commit message.</p> <p><span class="error">&#91;32&#93;</span> <a href="https://git.opendaylight.org/gerrit/#/c/60606/2" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/60606/2</a></p> <p>&gt; <span class="error">&#91;32&#93;</span> <a href="https://git.opendaylight.org/gerrit/#/c/60606/2" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/60606/2</a></p> <p>Not merged yet, Bug still present on releng <span class="error">&#91;33&#93;</span>.</p> <p><span class="error">&#91;33&#93;</span> <a href="https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/791/log.html.gz#s1-s32-t3-k2-k16-k1-k1" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/791/log.html.gz#s1-s32-t3-k2-k16-k1-k1</a></p> <p><span class="error">&#91;32&#93;</span> is merged, tests are mostly passing, but not 100%.</p> <p>Here is a recent releng failure: <span class="error">&#91;34&#93;</span>.</p> <p><span class="error">&#91;34&#93;</span> <a href="https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/794/log.html.gz#s1-s34-t7-k2-k16-k1-k1" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/794/log.html.gz#s1-s34-t7-k2-k16-k1-k1</a></p> <p>&gt; recent releng failure: <span class="error">&#91;34&#93;</span>.</p> <p>Oh, that happened for prefix-based shards, but before the fix those tests were passing (only module-based tests were failing on this Bug).</p> <p>So that may be a similar but different Bug. More investigation under way.</p> <p>&gt; More investigation</p> <p>Does not seem to be reproducible on Sandbox so far (runs 4-10 of <span class="error">&#91;35&#93;</span>),<br/> and (as usual for this Bug) nothing suspicious is visible on INFO level of releng logs.</p> <p>I suspect there is some inconsistency in the current implementation of produce-transactions which needs a specific timing to cause this symptom.</p> <p><span class="error">&#91;35&#93;</span> <a href="https://jenkins.opendaylight.org/sandbox/job/controller-csit-3node-clustering-ci-only-carbon/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/sandbox/job/controller-csit-3node-clustering-ci-only-carbon/</a></p> <p>&gt; Does not seem to be reproducible on Sandbox so far (runs 4-10 of <span class="error">&#91;35&#93;</span>)</p> <p>Run 15 succeeded at reproducing the failure <span class="error">&#91;36&#93;</span>.<br/> In the huge karaf.log <span class="error">&#91;37&#93;</span> I do not see anything suspicious before RequestTimeoutException makes it into AbstractTransactionHandler#txFailure, but then possibly something wrong happens, as there is nothing interesting logged after:</p> <p>2017-07-26 16:04:02,856 | DEBUG | pool-31-thread-2 | AbstractTransactionHandler | 257 - org.opendaylight.controller.samples.clustering-it-provider - 1.5.2.SNAPSHOT | org.opendaylight.controller.clustering.it.provider.impl.ProduceTransactionsHandler<br/> @bee1b2c Future #5634 failed in state RUNNING cause org.opendaylight.controller.cluster.access.client.RequestTimeoutException: Timed out after 120.024877455seconds</p> <p>Note that it means checkComplete has never been called, even though completigFuture has not been cancelled.<br/> I will try to reproduce with more logging.</p> <p><span class="error">&#91;36&#93;</span> <a href="https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-ci-only-carbon/15/log.html.gz#s1-s2-t3-k2-k16-k1-k1" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-ci-only-carbon/15/log.html.gz#s1-s2-t3-k2-k16-k1-k1</a><br/> <span class="error">&#91;37&#93;</span> <a href="https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-ci-only-carbon/15/odl2_karaf.log.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-ci-only-carbon/15/odl2_karaf.log.gz</a><br/> <span class="error">&#91;38&#93;</span> <a href="https://git.opendaylight.org/gerrit/#/c/60187/15" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/60187/15</a></p> <p>&gt; I will try to reproduce with more logging.</p> <p>Reproduced. This time the karaf.log <span class="error">&#91;39&#93;</span> shows that the progress stops when closing itemProducer. Will try to repeat with even more logs <span class="error">&#91;40&#93;</span>.</p> <p>2017-07-26 17:28:10,129 | TRACE | pool-31-thread-2 | ProduceTransactionsHandler | 257 - org.opendaylight.controller.samples.clustering-it-provider - 1.5.2.SNAPSHOT | org.opendaylight.controller.clustering.it.provider.impl.ProduceTransactionsHandler<br/> @30b30b0 Entering runFailed.</p> <p><span class="error">&#91;39&#93;</span> <a href="https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-ci-only-carbon/18/odl1_karaf.log.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-ci-only-carbon/18/odl1_karaf.log.gz</a><br/> 40https://git.opendaylight.org/gerrit/#/c/60783/1</p> <p>&gt; Will try to repeat with even more logs <span class="error">&#91;40&#93;</span>.</p> <p>No success reproducing with new trace logs yet.</p> <p>&gt; progress stops when closing itemProducer</p> <p>This feels different enough from the previous comments on this Bug, opened <a href="https://jira.opendaylight.org/browse/CONTROLLER-1745" title="produce-transactions can get stuck when closing itemProducer" class="issue-link" data-issue-key="CONTROLLER-1745"><del>CONTROLLER-1745</del></a>.</p> Blocks CONTROLLER-1700 Development External issue ID 8494 External issue URL Issue Type ODL SR Target Milestone Rank 0|i02s3z: