https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/CONTROLLER-1858 controller <p>DOMDataBroker Explicit Leader Movement test, controller-ddb-expl-lead-movement-longevity, has been failing fairly consistently <span class="error">&#91;1&#93;</span>. The purpose of test from test documentation:</p> <p>The goal is to ensure that applications do not observe disruption when a shard leader is moved as the result of explicit application request. This is performed by having a steady-stream producer execute operations against the shard and then initiate shard leader shutdown, then the producer is shut down cleanly.</p> <p>Though it shows 3/4 passing, those 3 are for restarting ODL with tell, actual test case has been failing consistently with only one pass on 28th July. The steps mentioned above are done repeatedly in a loop and it fails 1097 times during the entire ~14hour run with error 500 <span class="error">&#91;2&#93;</span>.</p> <p>Update:<br/> -----------<br/> Test steps detailed in <span class="error">&#91;3&#93;</span>.<br/> ODL Logs <span class="error">&#91;4&#93;</span></p> <p>As per <span class="error">&#91;3&#93;</span> searching ODL logs for 'starting leader movement from node' gives us:<br/> 018-08-04T23:40:26,930 | INFO | opendaylight-cluster-data-akka.actor.default-dispatcher-34 | ClientActorBehavior | 191 - org.opendaylight.controller.cds-access-client - 1.3.3.SNAPSHOT | member-1-frontend-datastore-config: connection ConnectedClientConnection{client=ClientIdentifier</p> {frontend=member-1-frontend-datastore-config, generation=0}, cookie=0, backend=ShardBackendInfo{actor=Actor<a href="#542053028" target="_blank" rel="noopener">akka://opendaylight-cluster-data/user/shardmanager-config/member-1-shard-default-config#542053028</a>, sessionId=0, version=BORON, maxMessages=1000, cookie=0, shard=default, dataTree=present}} indicated sequencing mismatch on member-1-datastore-config-fe-0-txn-13-0 sequence 0 (26), reconnecting it<br/> org.opendaylight.controller.cluster.access.commands.OutOfSequenceEnvelopeException: Expecting envelope 0<br/> at org.opendaylight.controller.cluster.datastore.LeaderFrontendState.checkRequestSequence(LeaderFrontendState.java:98) <span class="error">&#91;220:org.opendaylight.controller.sal-distributed-datastore:1.7.3.SNAPSHOT&#93;</span><br/> at org.opendaylight.controller.cluster.datastore.LeaderFrontendState.handleTransactionRequest(LeaderFrontendState.java:191) <span class="error">&#91;220:org.opendaylight.controller.sal-distributed-datastore:1.7.3.SNAPSHOT&#93;</span><br/> at org.opendaylight.controller.cluster.datastore.Shard.handleRequest(Shard.java:573) <span class="error">&#91;220:org.opendaylight.controller.sal-distributed-datastore:1.7.3.SNAPSHOT&#93;</span><br/> at org.opendaylight.controller.cluster.datastore.Shard.handleRequestEnvelope(Shard.java:398) <span class="error">&#91;220:org.opendaylight.controller.sal-distributed-datastore:1.7.3.SNAPSHOT&#93;</span><br/> <br/> Also:<br/> <br/> 2018-08-05T01:17:16,637 | INFO | opendaylight-cluster-data-akka.actor.default-dispatcher-36 | ClientActorBehavior | 191 - org.opendaylight.controller.cds-access-client - 1.3.3.SNAPSHOT | member-1-frontend-datastore-config: connection ConnectedClientConnection{client=ClientIdentifier{frontend=member-1-frontend-datastore-config, generation=0} <p>, cookie=0, backend=ShardBackendInfo{actor=Actor<span class="error">&#91;akka.tcp://opendaylight-cluster-data@10.30.170.171:2550/user/shardmanager-config/member-2-shard-default-config#-1686300808&#93;</span>, sessionId=71, version=BORON, maxMessages=1000, cookie=0, shard=default, dataTree=absent}} indicated no leadership, reconnecting it<br/> org.opendaylight.controller.cluster.access.commands.NotLeaderException: Actor Actor<a href="#-1686300808" target="_blank" rel="noopener">akka://opendaylight-cluster-data/user/shardmanager-config/member-2-shard-default-config#-1686300808</a> is not the current leader<br/> at org.opendaylight.controller.cluster.datastore.Shard.handleRequest(Shard.java:566) ~<span class="error">&#91;220:org.opendaylight.controller.sal-distributed-datastore:1.7.3.SNAPSHOT&#93;</span><br/> at org.opendaylight.controller.cluster.datastore.Shard.handleRequestEnvelope(Shard.java:398) ~<span class="error">&#91;220:org.opendaylight.controller.sal-distributed-datastore:1.7.3.SNAPSHOT&#93;</span><br/> at org.opendaylight.controller.cluster.datastore.Shard.handleNonRaftCommand(Shard.java:324) ~<span class="error">&#91;220:org.opendaylight.controller.sal-distributed-datastore:1.7.3.SNAPSHOT&#93;</span></p> <p><span class="error">&#91;1&#93;</span> <a href="https://jenkins.opendaylight.org/releng/job/controller-csit-3node-ddb-expl-lead-movement-longevity-only-oxygen/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/job/controller-csit-3node-ddb-expl-lead-movement-longevity-only-oxygen/</a><br/> <span class="error">&#91;2&#93;</span> <a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/controller-csit-3node-ddb-expl-lead-movement-longevity-only-oxygen/51/robot-plugin/log.html.gz#s1-s2-t1-k2-k1-k1-k1-k1-k1-k1-k1" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/controller-csit-3node-ddb-expl-lead-movement-longevity-only-oxygen/51/robot-plugin/log.html.gz#s1-s2-t1-k2-k1-k1-k1-k1-k1-k1-k1</a><br/> <span class="error">&#91;3&#93;</span> <a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/controller-csit-3node-ddb-expl-lead-movement-longevity-only-oxygen/51/robot-plugin/log.html.gz#s1-s2-t1-k2-k1-k1-k1-k1-k1-k1-k1" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/controller-csit-3node-ddb-expl-lead-movement-longevity-only-oxygen/51/robot-plugin/log.html.gz#s1-s2-t1-k2-k1-k1-k1-k1-k1-k1-k1</a><br/> <span class="error">&#91;4&#93;</span> <a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/controller-csit-3node-ddb-expl-lead-movement-longevity-only-oxygen/51/" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/controller-csit-3node-ddb-expl-lead-movement-longevity-only-oxygen/51/</a></p> CONTROLLER-1858

DOMDataBroker Explicit Leader Movement test fails

Bug High Open Unresolved Vishal Thapar Vishal Thapar csit:3node Tue, 7 Aug 2018 07:53:17 +0000 Fri, 17 Aug 2018 18:01:31 +0000 Oxygen SR3 clustering 0 4 <p>What are the exact steps this test performs and what failed? I see "FAIL [ 200 | 201 | 204 ] does not contain value '500'." but I don't see from the robot log what URL failed and the error message.</p> <p>I think it would really help to outline the test steps in sentences for those not all that familiar with all the keywords and the author's intent - I can ascertain/infer some meaning by looking at the log output (perhaps this is already documented somewhere?) but I don't know what all the keywords do/mean. The first red output I see is:</p> <p> <br/> KEYWORD BuiltIn . Fail ${output}<br/>  <br/>  </p> <div class='table-wrap'> <table class='confluenceTable'><tbody> <tr> <th class='confluenceTh'>Documentation:</th> <td class='confluenceTd'>Fails the test with the given message and optionally alters its tags.</td> </tr> <tr> <th class='confluenceTh'>Start / End / Elapsed:</th> <td class='confluenceTd'>20180805 09:36:16.606 / 20180805 09:36:16.607 / 00:00:00.001</td> </tr> <tr> <td class='confluenceTd'>09:36:16.607</td> <td class='confluenceTd'>FAIL</td> <td class='confluenceTd'>None</td> </tr> </tbody></table> </div> <p> </p> <p>But I can't tell what actually failed - the prior URL,  /restconf/operations/cluster-admin:make-leader-local, succeeded.</p> <p> </p> <p>Also is this Oxygen or Fluorine or both? Can you please update the Affected Version? If this is from Oxygen, does it also fail on Fluorine?</p> <p>Agree, I also been trying to decipher controller tests and thought you might be able to understand them better than me. Have to really dig into robot keywords to get to useful information.</p> <p>What my understanding of the test is it is doing following:</p> <p>1. Move leader from nodeA to a randomly chosen nodeB.</p> <p>2. Write transaction using RPCs - <a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/controller-csit-3node-ddb-expl-lead-movement-longevity-only-oxygen/51/robot-plugin/log.html.gz#s1-s2-t1-k2-k1-k1-k1-k1-k1-k1-k1-k1-k1-k2-k5" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/controller-csit-3node-ddb-expl-lead-movement-longevity-only-oxygen/51/robot-plugin/log.html.gz#s1-s2-t1-k2-k1-k1-k1-k1-k1-k1-k1-k1-k1-k2-k5</a></p> <p>3. Wait for 5 seconds.</p> <p>4. Check if all transactions went through fine or not. It checks for http code to be <span class="error">&#91;200 | 201 | 204&#93;</span>. But finds a 500.</p> <p>5. So, basically while doing a write transaction when leader movement is done, it gets http error 500.</p> <p>This step in robot:<br/> <a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/controller-csit-3node-ddb-expl-lead-movement-longevity-only-oxygen/51/robot-plugin/log.html.gz#s1-s2-t1-k2-k1-k1-k1-k1-k1-k1-k1-k1-k1-k2-k2" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/controller-csit-3node-ddb-expl-lead-movement-longevity-only-oxygen/51/robot-plugin/log.html.gz#s1-s2-t1-k2-k1-k1-k1-k1-k1-k1-k1-k1-k1-k2-k2</a></p> <p>logs the log message in karaf.</p> <p>This one is oxygen, I believe we don't have a flourine job for it yet.</p> <p>So the question is what URL failed with 500 and what was the error info? </p> <p>I thought I pasted this:</p> <p><a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/controller-csit-3node-ddb-expl-lead-movement-longevity-only-oxygen/51/robot-plugin/log.html.gz#s1-s2-t1-k2-k1-k1-k1-k1-k1-k1-k1-k1-k1-k2-k5" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/controller-csit-3node-ddb-expl-lead-movement-longevity-only-oxygen/51/robot-plugin/log.html.gz#s1-s2-t1-k2-k1-k1-k1-k1-k1-k1-k1-k1-k1-k2-k5</a></p> <p>Documentation: <br/> Invoke write-transactions rpc on given nodes.</p> <p>Start / End / Elapsed: 20180805 19:05:36.555 / 20180805 19:05:36.558 / 00:00:00.003<br/> 19:05:36.556 INFO Input parameters: host_list:<span class="error">&#91;u&#39;10.30.170.171&#39;&#93;</span>, index_list:<span class="error">&#91;2&#93;</span>, id_prefix:prefix-, duration:40, rate:1000, chained_flag:True <br/> 19:05:36.556 INFO url: <a href="http://10.30.170.171:8181/restconf/operations/odl-mdsal-lowlevel-control:write-transactions" class="external-link" target="_blank" rel="nofollow noopener">http://10.30.170.171:8181/restconf/operations/odl-mdsal-lowlevel-control:write-transactions</a>, data: &lt;input xmlns="tag:opendaylight.org,2017:controller:yang:lowlevel:control"&gt;<br/> &lt;id&gt;prefix-2&lt;/id&gt;<br/> &lt;seconds&gt;40&lt;/seconds&gt;<br/> &lt;transactions-per-second&gt;1000&lt;/transactions-per-second&gt;<br/> &lt;chained-transactions&gt;True&lt;/chained-transactions&gt;<br/> &lt;/input&gt;, timeout: 425</p> <p>From the url, it looks like test app is running in ODL and RPC just triggers it. Correct? That is why couldn't find logs for any python script making rest calls. Suspect some test app within controller code that generates transactions when it gets this RPC call.</p> <p>yeah - it looks like, restconf/operations/odl-mdsal-lowlevel-control:write-transactions.  However AFAICT, the response from that was 200:</p> <p> </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>KEYWORD BuiltIn . Log ${response.text}   ||Documentation:|Logs the given message with the given level.| ||Start / End / Elapsed:|20180805 09:36:16.600 / 20180805 09:36:16.600 / 00:00:00.000| |09:36:16.600|INFO|{"output":\{"delete-tx":731,"all-tx":40000,"insert-tx":39269}}| | 00:00:00.001KEYWORD BuiltIn .  Log ${response.status_code} ||Documentation:|Logs the given message with the given level.| ||Start / End / Elapsed:|20180805 09:36:16.600 / 20180805 09:36:16.601 / 00:00:00.001| |09:36:16.601|INFO|200| </pre> </div></div> <p>The encapsulating keyword seems to be OK as well, AFAICT:</p> <p> </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>09:36:16.604 INFO ${status} = PASS 09:36:16.604 INFO ${output} = None   </pre> </div></div> <p>The code for that RPC would be somewhere in the clustering-it-provider in controller. </p> <p>So I don't see where the 500 is coming from or the nature of this failure:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>09:36:16.607 FAIL None </pre> </div></div> <p>I think it is this one.</p> <p><a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/controller-csit-3node-ddb-expl-lead-movement-longevity-only-oxygen/51/robot-plugin/log.html.gz#s1-s2-t1-k2-k1-k1-k1-k1-k1-k1-k2-k1-k1-k2-k10" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/controller-csit-3node-ddb-expl-lead-movement-longevity-only-oxygen/51/robot-plugin/log.html.gz#s1-s2-t1-k2-k1-k1-k1-k1-k1-k1-k2-k1-k1-k2-k10</a></p> <p>OK - so the underlying error that caused the 500 occurred on odl2:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>2018-08-05T13:36:22,354 | WARN | opendaylight-cluster-data-shard-dispatcher-42 | AbstractFrontendHistory | 220 - org.opendaylight.controller.sal-distributed-datastore - 1.7.3.SNAPSHOT | member-2-shard-default-config: Request ModifyTransactionRequest{target=member-1-datastore-config-fe-0-txn-1114-0, sequence=0, replyTo=Actor[akka.tcp://opendaylight-cluster-data@10.30.170.146:2550/user/$a#-1538190485], modifications=1, protocol=SIMPLE} is contained purged transactions [[0..1114]] TransactionCommitFailedException{message=canCommit encountered an unexpected failure, errorList=[RpcError [message=canCommit encountered an unexpected failure, severity=ERROR, errorType=APPLICATION, tag=operation-failed, applicationTag=null, info=null, cause=org.opendaylight.controller.cluster.access.commands.DeadTransactionException: Transactions [[0..1114]] have been purged]]} org.opendaylight.controller.md.sal.dom.broker.impl.TransactionCommitFailedExceptionMapper.newWithCause(TransactionCommitFailedExceptionMapper.java:36) org.opendaylight.controller.md.sal.dom.broker.impl.TransactionCommitFailedExceptionMapper.newWithCause(TransactionCommitFailedExceptionMapper.java:18) org.opendaylight.yangtools.util.concurrent.ExceptionMapper.apply(ExceptionMapper.java:106)\n\tat org.opendaylight.controller.cluster.databroker.ConcurrentDOMDataBroker.handleException(ConcurrentDOMDataBroker.java:227) org.opendaylight.controller.cluster.databroker.ConcurrentDOMDataBroker.access$000(ConcurrentDOMDataBroker.java:50) org.opendaylight.controller.cluster.databroker.ConcurrentDOMDataBroker$1.onFailure(ConcurrentDOMDataBroker.java:124) com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1387) com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:398) com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1015) com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:868) com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:713) com.google.common.util.concurrent.SettableFuture.setException(SettableFuture.java:54) org.opendaylight.controller.cluster.databroker.actors.dds.AbstractProxyTransaction.lambda$directCommit$4(AbstractProxyTransaction.java:507) org.opendaylight.controller.cluster.access.client.ConnectionEntry.complete(ConnectionEntry.java:50) org.opendaylight.controller.cluster.access.client.AbstractClientConnection.receiveResponse(AbstractClientConnection.java:469) org.opendaylight.controller.cluster.access.client.AbstractReceivingClientConnection.receiveResponse(AbstractReceivingClientConnection.java:68) org.opendaylight.controller.cluster.access.client.ClientActorBehavior.onResponse(ClientActorBehavior.java:174) org.opendaylight.controller.cluster.access.client.ClientActorBehavior.onRequestFailure(ClientActorBehavior.java:186) org.opendaylight.controller.cluster.access.client.ClientActorBehavior.internalOnRequestFailure(ClientActorBehavior.java:238) org.opendaylight.controller.cluster.access.client.ClientActorBehavior.onReceiveCommand(ClientActorBehavior.java:145) org.opendaylight.controller.cluster.access.client.ClientActorBehavior.onReceiveCommand(ClientActorBehavior.java:49) org.opendaylight.controller.cluster.access.client.AbstractClientActor.onReceiveCommand(AbstractClientActor.java:69) akka.persistence.UntypedPersistentActor.onReceive(PersistentActor.scala:275) akka.actor.UntypedActor$$anonfun$receive$1.applyOrElse(UntypedActor.scala:167) akka.actor.Actor.aroundReceive(Actor.scala:517) akka.actor.Actor.aroundReceive$(Actor.scala:515) akka.persistence.UntypedPersistentActor.akka$persistence$Eventsourced$$super$aroundReceive(PersistentActor.scala:273) akka.persistence.Eventsourced$$anon$1.stateReceive(Eventsourced.scala:691) akka.persistence.Eventsourced.aroundReceive(Eventsourced.scala:192) akka.persistence.Eventsourced.aroundReceive$(Eventsourced.scala:191) akka.persistence.UntypedPersistentActor.aroundReceive(PersistentActor.scala:273) akka.actor.ActorCell.receiveMessage(ActorCell.scala:590) akka.actor.ActorCell.invoke(ActorCell.scala:559) akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257) akka.dispatch.Mailbox.run(Mailbox.scala:224) akka.dispatch.Mailbox.exec(Mailbox.scala:234) akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) Caused by: org.opendaylight.controller.cluster.access.commands.DeadTransactionException: Transactions [[0..1114]] have been purged org.opendaylight.controller.cluster.datastore.AbstractFrontendHistory.handleTransactionRequest(AbstractFrontendHistory.java:88) org.opendaylight.controller.cluster.datastore.LeaderFrontendState.handleTransactionRequest(LeaderFrontendState.java:212) org.opendaylight.controller.cluster.datastore.Shard.handleRequest(Shard.java:573) org.opendaylight.controller.cluster.datastore.Shard.handleRequestEnvelope(Shard.java:398) org.opendaylight.controller.cluster.datastore.Shard.handleNonRaftCommand(Shard.java:324) org.opendaylight.controller.cluster.raft.RaftActor.handleCommand(RaftActor.java:270) org.opendaylight.controller.cluster.common.actor.AbstractUntypedPersistentActor.onReceiveCommand(AbstractUntypedPersistentActor.java:44) akka.persistence.UntypedPersistentActor.onReceive(PersistentActor.scala:275) org.opendaylight.controller.cluster.common.actor.MeteringBehavior.apply(MeteringBehavior.java:104) </pre> </div></div> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=rovarga" class="user-hover" rel="rovarga">rovarga</a> any ideas on this one?</p> <p>I think we're going to have to enabled some debug. Set <em>org.opendaylight.controller.cluster.datastore</em> and <em>org.opendaylight.controller.cluster.databroker</em> to DEBUG. This will also include Shard which will emit a lot of output so to minimize it, set <em>org.opendaylight.controller.cluster.datastore.Shard</em> to INFO. This should do it:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>log4j2.logger.datastore.name=org.opendaylight.controller.cluster.datastore log4j2.logger.datastore.level=DEBUG log4j2.logger.databroker.name=org.opendaylight.controller.cluster.databroker log4j2.logger.databroker.level=DEBUG log4j2.logger.shard.name=org.opendaylight.controller.cluster.datastore.Shard log4j2.logger.shard.level=INFO </pre> </div></div> <p>Note: the log files may roll over, ie there may also be karaf.log.1, karaf.log.2 etc. </p> <p>this <a href="https://jenkins.opendaylight.org/releng/user/jluhrsen/my-views/view/controller%203node/job/controller-csit-3node-ddb-expl-lead-movement-longevity-only-oxygen/51" class="external-link" target="_blank" rel="nofollow noopener">job </a> will be running with the above debug levels</p> <p><a href="https://jenkins.opendaylight.org/releng/job/controller-csit-3node-ddb-expl-lead-movement-longevity-only-oxygen/52/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/job/controller-csit-3node-ddb-expl-lead-movement-longevity-only-oxygen/52/</a> </p> <p>^Passed with deubg</p> <p><a href="https://jenkins.opendaylight.org/releng/job/controller-csit-3node-ddb-expl-lead-movement-longevity-only-oxygen/53/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/job/controller-csit-3node-ddb-expl-lead-movement-longevity-only-oxygen/53/</a></p> <p>^Had failures but debugs were not enabled, probably scheduled run.</p> <p>trying again:<br/> <a href="https://jenkins.opendaylight.org/releng/job/controller-csit-3node-ddb-expl-lead-movement-longevity-only-oxygen/54/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/job/controller-csit-3node-ddb-expl-lead-movement-longevity-only-oxygen/54/</a></p> <p>This one got failure with debug logs:</p> <p><a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/controller-csit-3node-ddb-expl-lead-movement-longevity-only-oxygen/54/" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/controller-csit-3node-ddb-expl-lead-movement-longevity-only-oxygen/54/</a></p> <p>Warning: Log files are ~10/19/19MB compressed. Would recommend doing a wget to get them and then untar them.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>. Fail ${output} Documentation: Fails the test with the given message and optionally alters its tags. Start / End / Elapsed: 20180816 08:36:31.438 / 20180816 08:36:31.439 / 00:00:00.001 08:36:31.439 FAIL Dictionary does not contain key 'value'. </pre> </div></div> <p>This is a different error. A jolokia request failed:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>To JSON using : content={ "error": "javax.management.InstanceNotFoundException : org.opendaylight.controller:Category=Shards,name=member-2-shard-default-config,type=DistributedConfigDatastore", "error_type": "javax.management.InstanceNotFoundException", "request": { "mbean": "org.opendaylight.controller:Category=Shards,name=member-2-shard-default-config,type=DistributedConfigDatastore", "type": "read" }, "stacktrace": "javax.management.InstanceNotFoundException: org.opendaylight.controller:Category=Shards,name=member-2-shard-default-config,type=DistributedConfigDatastore\n\tat com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1095)\n\tat com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBeanInfo(DefaultMBeanServerInterceptor.java:1375)\n\tat com.sun.jmx.mbeanserver.JmxMBeanServer.getMBeanInfo(JmxMBeanServer.java:920)\n\tat org.jolokia.handler.ReadHandler$1.execute(ReadHandler.java:46)\n\tat org.jolokia.handler.ReadHandler$1.execute(ReadHandler.java:41)\n\tat org.jolokia.backend.executor.AbstractMBeanServerExecutor.call(AbstractMBeanServerExecutor.java:90)\n\tat org.jolokia.handler.ReadHandler.getMBeanInfo(ReadHandler.java:229)\n\tat org.jolokia.handler.ReadHandler.getAllAttributesNames(ReadHandler.java:241)\n\tat org.jolokia.handler.ReadHandler.resolveAttributes(ReadHandler.java:217)\n\tat org.jolokia.handler.ReadHandler.fetchAttributes(ReadHandler.java:183)\n\tat org.jolokia.handler.ReadHandler.doHandleRequest(ReadHandler.java:118)\n\tat org.jolokia.handler.ReadHandler.doHandleRequest(ReadHandler.java:37)\n\tat org.jolokia.handler.JsonRequestHandler.handleRequest(JsonRequestHandler.java:161)\n\tat org.jolokia.backend.MBeanServerHandler.dispatchRequest(MBeanServerHandler.java:156)\n\tat org.jolokia.backend.LocalRequestDispatcher.dispatchRequest(LocalRequestDispatcher.java:99)\n\tat org.jolokia.backend.BackendManager.callRequestDispatcher(BackendManager.java:428)\n\tat org.jolokia.backend.BackendManager.handleRequest(BackendManager.java:158)\n\tat org.jolokia.http.HttpRequestHandler.executeRequest(HttpRequestHandler.java:197)\n\tat org.jolokia.http.HttpRequestHandler.handleGetRequest(HttpRequestHandler.java:86)\n\tat org.jolokia.http.AgentServlet$4.handleRequest(AgentServlet.java:470)\n\tat org.jolokia.http.AgentServlet.handleSecurely(AgentServlet.java:350)\n\tat org.jolokia.http.AgentServlet.handle(AgentServlet.java:321)\n\tat org.jolokia.http.AgentServlet.doGet(AgentServlet.java:277)\n\tat javax.servlet.http.HttpServlet.service(HttpServlet.java:687)\n\tat javax.servlet.http.HttpServlet.service(HttpServlet.java:790)\n\tat org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848)\n\tat org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1772)\n\tat org.eclipse.jetty.websocket.server.WebSocketUpgradeFilter.doFilter(WebSocketUpgradeFilter.java:205)\n\tat org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)\n\tat org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)\n\tat org.ops4j.pax.web.service.jetty.internal.HttpServiceServletHandler.doHandle(HttpServiceServletHandler.java:71)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)\n\tat org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)\n\tat org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)\n\tat org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)\n\tat org.ops4j.pax.web.service.jetty.internal.HttpServiceContext.doHandle(HttpServiceContext.java:284)\n\tat org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)\n\tat org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)\n\tat org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)\n\tat org.ops4j.pax.web.service.jetty.internal.JettyServerHandlerCollection.handle(JettyServerHandlerCollection.java:80)\n\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)\n\tat org.eclipse.jetty.server.Server.handle(Server.java:534)\n\tat org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333)\n\tat org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)\n\tat org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)\n\tat org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)\n\tat org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)\n\tat org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)\n\tat org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)\n\tat org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)\n\tat java.lang.Thread.run(Thread.java:748)\n", "status": 404 } </pre> </div></div> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>2018-08-16T12:36:11,723 | ERROR | opendaylight-cluster-data-akka.actor.default-dispatcher-4 | Shard | 213 - org.opendaylight.controller.sal-clustering-commons - 1.7.4.SNAPSHOT | Failed to persist event type [org.opendaylight.controller.cluster.raft.persisted.SimpleReplicatedLogEntry] with sequence number [31300240] for persistenceId [member-2-shard-default-config]. akka.pattern.CircuitBreaker$$anon$1: Circuit Breaker Timed out. </pre> </div></div> <p>akka timed out trying to write to disk. This indicates the disk was busy - probably due to all the logging. We'll have to increase the timeout - I'll need to look that up and report back later. </p> <p>thanks <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=tpantelis" class="user-hover" rel="tpantelis">tpantelis</a>, I'm assuming this timeout increase has to come in code, and not some config file?<br/> in the meantime, I removed the cluster.datastore DEBUG logging and just left these:</p> <p>controller.cluster.databroker:DEBUG<br/> controller.cluster.datastore.Shard:INFO</p> <p>running in <a href="https://jenkins.opendaylight.org/releng/job/controller-csit-3node-ddb-expl-lead-movement-longevity-only-oxygen/55" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/job/controller-csit-3node-ddb-expl-lead-movement-longevity-only-oxygen/55</a></p> <p>It's an akka setting:</p> <p>In controller/configuration/initial/akka.conf, set the timeout to like 5 min:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>akka { persistence { journal-plugin-fallback { circuit-breaker { call-timeout = 300s } } snapshot-store-plugin-fallback { circuit-breaker { max-failures = 10 call-timeout = 300s } } } } </pre> </div></div> <p>We need cluster.datastore DEBUG.</p> Development Rank 0|i03hp3: