https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/CONTROLLER-1789 controller <p>Under heavy transaction load in cluster setup, sometimes akka persistence circuit-breaker failure is observed. When this happens, the shard gets stopped and it never recovers from this condition. This bug is opened to request recovery of shard when it gets into this condition</p> <p>Problem can be forced to happen pretty easily:<br/> 1. On node-1, install controller in a nfs mounted directory<br/> 2. Start all nodes and make sure node-1 is leader of the target shard<br/> 3. Make write transactions at high rate against this shard<br/> 4. Stop/start the nfs service on server from which the dir was mounted. This will cause the dir to be not writable for some time<br/> 5. During this period, akka persistence for journal records will fail. When this failure happens, the shard is stopped and never started again</p> <p>Increasing the akka journal persistence circuit-breaker call-timeout value (default is 10s) does help in making it more tolerant to outage</p> <p>Few others have seen this issue. Ref. <a href="https://lists.opendaylight.org/pipermail/controller-dev/2017-August/013777.html" class="external-link" target="_blank" rel="nofollow noopener">https://lists.opendaylight.org/pipermail/controller-dev/2017-August/013777.html</a></p> CONTROLLER-1789

Recover from shard stopped condition on akka persistence circuit-breaker failure

Bug Medium Confirmed Unresolved Unassigned Ajay Lele Tue, 14 Nov 2017 20:07:11 +0000 Tue, 18 May 2021 15:35:59 +0000 0 2 <p>The basic cause of Circuit Breaker issues is that disk I/O operation takes longer than the (circuit-breaker) timeouts used by akka persistence plugins. This could be because disk is slow/choked, amount of data in data-store is very large (order of hundreds of MBs), transactions are happening at very high rate, or a combination of these factors. Suggested steps for troubleshooting this problem:</p> <ol> <li>Monitor disk performance by using tools like sar <span class="error">&#91;0&#93;</span>, identify and fix bottle-necks</li> <li>Tune akka persistence plugin circuit-breaker settings. In controller/configuration/initial/akka.conf, override the defaults, esp. the call-timeout. Refer to <span class="error">&#91;1&#93;</span> and <span class="error">&#91;2&#93;</span> for more details. Example:</li> </ol> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>akka { persistence { journal-plugin-fallback { circuit-breaker { max-failures = 10 call-timeout = 60s reset-timeout = 30s } } snapshot-store-plugin-fallback { circuit-breaker { max-failures = 10 call-timeout = 120s reset-timeout = 60s } } } }</pre> </div></div> <p><span class="error">&#91;0&#93;</span> <a href="https://linux.die.net/man/1/sar" class="external-link" target="_blank" rel="nofollow noopener">https://linux.die.net/man/1/sar</a></p> <p><span class="error">&#91;1&#93;</span> <a href="https://github.com/akka/akka/blob/master/akka-persistence/src/main/resources/reference.conf" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/akka/akka/blob/master/akka-persistence/src/main/resources/reference.conf</a></p> <p><span class="error">&#91;2&#93;</span> <a href="https://doc.akka.io/docs/akka/2.5/common/circuitbreaker.html" class="external-link" target="_blank" rel="nofollow noopener">https://doc.akka.io/docs/akka/2.5/common/circuitbreaker.html</a></p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=ajayslele" class="user-hover" rel="ajayslele">ajayslele</a> any progress on this? Are you still working on it?</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=rovarga" class="user-hover" rel="rovarga">rovarga</a> yes, I will be proposing a patch soon</p> <p>The PR which I had opened for this <span class="error">&#91;0&#93;</span> was in pretty good shape, but could not get through the reviews. Whoever wants to work on this can pick up from where it is right now.</p> <p><span class="error">&#91;0&#93;</span> <a href="https://git.opendaylight.org/gerrit/c/controller/+/79328" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/c/controller/+/79328</a></p> Development Rank 0|i037qf: