https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/CONTROLLER-1866 controller <p>There is a failure showing up in netvirt 1node CSIT where the output of the karaf<br/> cli "showSvcStatus" has the DATASTORE in ERROR state:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> Timestamp: Tue Oct 09 21:02:42 UTC 2018 Node IP Address: 10.30.170.157 <span class="code-object">System</span> is operational: <span class="code-keyword">false</span> <span class="code-object">System</span> ready state: ACTIVE OPENFLOW : OPERATIONAL IFM : OPERATIONAL ITM : OPERATIONAL ELAN : OPERATIONAL OVSDB : OPERATIONAL DATASTORE : ERROR java.lang.reflect.UndeclaredThrowableException </pre> </div></div> <p>Looking at the <a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-0cmb-1ctl-2cmp-openstack-queens-upstream-stateful-snat-conntrack-oxygen/86/odl_1/odl1_karaf.log.gz" class="external-link" target="_blank" rel="nofollow noopener">karaf.log </a> it seems the reason for this is that we hit<br/> a circuit breaker timed out issue and some cluster/akka logic is shutting down<br/> the datastore.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> 2018-10-09T20:58:22,469 | ERROR | opendaylight-cluster-data-akka.actor.<span class="code-keyword">default</span>-dispatcher-39 | Shard | 228 - org.opendaylight.controller.sal-clustering-commons - 1.7.4 | Failed to persist event type [org.opendaylight.controller.cluster.raft.persisted.SimpleReplicatedLogEntry] with sequence number [78318] <span class="code-keyword">for</span> persistenceId [member-1-shard-<span class="code-keyword">default</span>-config]. akka.pattern.CircuitBreaker$$anon$1: Circuit Breaker Timed out. 2018-10-09T20:58:22,515 | INFO | opendaylight-cluster-data-shard-dispatcher-215 | Shard | 228 - org.opendaylight.controller.sal-clustering-commons - 1.7.4 | Stopping Shard member-1-shard-<span class="code-keyword">default</span>-config 2018-10-09T20:58:22,517 | WARN | opendaylight-cluster-data-akka.actor.<span class="code-keyword">default</span>-dispatcher-70 | LocalThreePhaseCommitCohort | 235 - org.opendaylight.controller.sal-distributed-datastore - 1.7.4 | Failed to prepare transaction member-1-datastore-config-fe-0-txn-65215-0 on backend java.lang.RuntimeException: Transaction aborted due to shutdown. </pre> </div></div> <p>This is not neccessarily a heavy job so I am not suspecting that this job is not able<br/> to keep up with writing to disk, which I think is one reason this might happen.<br/>  </p> CONTROLLER-1866

circuit breaker timed out; datastore shutdown

Bug Medium Resolved Cannot Reproduce Jamo Luhrsen Jamo Luhrsen csit csit:failures Wed, 17 Oct 2018 05:32:21 +0000 Sat, 20 Oct 2018 04:46:22 +0000 Sat, 20 Oct 2018 04:46:22 +0000 Neon Fluorine SR1 Oxygen SR4 clustering 0 2 <p>This failure usually means the disk is busy/overloaded or slow. The CSIT env uses shared disks I assume so perhaps there were other jobs or whatever running using the disk heavily. It could also be due in part to a really high rate of transactions. </p> <p>The sequence number was 78318 which seems pretty high, especially for the default config shard. You say it's not a heavy job but that seems a bit heavy although I don't know how long a period the job runs for or what it does. If this failure is happening repeatedly now all of a sudden then I would suggest looking into the app code or the test - perhaps some recent change is causing an unusually high transaction rate?</p> <p>Either way the only potential remedy to alleviate the failure from the CDS or akka or levelDB side is to increase the timeout (I think the default timeout is 5 sec). In controller/configuration/initial/akka.conf,</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>akka { persistence { journal-plugin-fallback { circuit-breaker { call-timeout = 300s } } snapshot-store-plugin-fallback { circuit-breaker { max-failures = 10 call-timeout = 300s } } } } </pre> </div></div> <p>ok, I searched through ~300 job results and this was the only instance of this failure. Let's go with the theory that this<br/> was caused by some external processes in this shared environment consuming/overloading the disk. I'll close it out.</p> <p>assuming to be caused by an overloaded environment</p> Development Rank 0|i03jpz: