https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/CONTROLLER-1755 controller <p>The fix for <a href="https://jira.opendaylight.org/browse/CONTROLLER-1746" title="OOM with large number of closed transactions" class="issue-link" data-issue-key="CONTROLLER-1746"><del>CONTROLLER-1746</del></a> exposes a problem with RaftActor, where the sequencing of applied state is violated and the same journal entry is being applied &#8211; leading to violation of transaction ordering in DataTree.</p> <p>This is readily reproduced in <a href="https://jenkins.opendaylight.org/releng/view/netvirt/job/netvirt-csit-3node-openstack-ocata-gate-stateful-carbon/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/netvirt/job/netvirt-csit-3node-openstack-ocata-gate-stateful-carbon/</a> and we saw this happening with Cluster Singleton Service &#8211; at high transition rates EOS blows up on the same problem.</p> <p>Operating System: All<br/> Platform: All</p> CONTROLLER-1755

RaftActor lastApplied index moves backwards

Bug Resolved Done Robert Varga Robert Varga Tue, 22 Aug 2017 14:10:42 +0000 Wed, 6 Sep 2017 16:35:38 +0000 Wed, 6 Sep 2017 16:35:38 +0000 Carbon clustering 0 3 <p>Attachment csit.txt has been added with description: CSIT failure log</p> <p>prototype fix: <a href="https://git.opendaylight.org/gerrit/#/c/62107/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/62107/</a></p> <p>Blocks <a href="https://jira.opendaylight.org/browse/MDSAL-275" title="VerifyException from flapping service singleton registration" class="issue-link" data-issue-key="MDSAL-275"><del>MDSAL-275</del></a> as the scale test for it is triggering this bug, as seen in <a href="https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-cs-chasing-leader-longevity-only-carbon/7/odl2_karaf.log.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-cs-chasing-leader-longevity-only-carbon/7/odl2_karaf.log.gz</a>.</p> <p><a href="https://jenkins.opendaylight.org/releng/view/netvirt/job/netvirt-csit-3node-openstack-ocata-gate-stateful-carbon/7&#39;s" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/netvirt/job/netvirt-csit-3node-openstack-ocata-gate-stateful-carbon/7's</a> odl3_karaf.log shows the same failure at around 2017-08-21 22:50:10,355, along with the time travel:</p> <p>2017-08-21 22:50:10,355 | DEBUG | rd-dispatcher-34 | Shard | 196 - org.opendaylight.controller.sal-clustering-commons - 1.5.2.SNAPSHOT | member-3-shard-default-operational (Leader): Replicate message: identifier: member-3-datastore-operational-fe-1-txn-1133-0, logIndex: 1943, payload: class org.opendaylight.controller.cluster.datastore.persisted.PurgeTransactionPayload, isSendImmediate: false<br/> 2017-08-21 22:50:10,355 | DEBUG | rd-dispatcher-34 | Shard | 196 - org.opendaylight.controller.sal-clustering-commons - 1.5.2.SNAPSHOT | member-3-shard-default-operational (Leader): Applying to log - commitIndex: 1940, lastAppliedIndex: 1939<br/> 2017-08-21 22:50:10,355 | DEBUG | rd-dispatcher-34 | Shard | 196 - org.opendaylight.controller.sal-clustering-commons - 1.5.2.SNAPSHOT | member-3-shard-default-operational (Leader): Setting last applied to 1940<br/> 2017-08-21 22:50:10,355 | DEBUG | rd-dispatcher-34 | Shard | 196 - org.opendaylight.controller.sal-clustering-commons - 1.5.2.SNAPSHOT | member-3-shard-default-operational: Moving last applied index from 1939 to 1940<br/> 2017-08-21 22:50:10,355 | DEBUG | rd-dispatcher-34 | Shard | 196 - org.opendaylight.controller.sal-clustering-commons - 1.5.2.SNAPSHOT | member-3-shard-default-operational: Applying state for log index 1940 data org.opendaylight.controller.cluster.datastore.persisted.PurgeTransactionPayload@74db3f62<br/> 2017-08-21 22:50:10,355 | DEBUG | rd-dispatcher-34 | ShardDataTree | 203 - org.opendaylight.controller.sal-distributed-datastore - 1.5.2.SNAPSHOT | member-3-shard-default-operational: replication of member-3-datastore-operational-fe-1-txn-1131-0 has no callback<br/> 2017-08-21 22:50:10,362 | DEBUG | rd-dispatcher-34 | Shard | 196 - org.opendaylight.controller.sal-clustering-commons - 1.5.2.SNAPSHOT | member-3-shard-default-operational (Leader): Setting last applied to 1940<br/> 2017-08-21 22:50:10,362 | DEBUG | rd-dispatcher-34 | Shard | 196 - org.opendaylight.controller.sal-clustering-commons - 1.5.2.SNAPSHOT | member-3-shard-default-operational: Moving last applied index from 1940 to 1940<br/> 2017-08-21 22:50:10,362 | DEBUG | rd-dispatcher-34 | Shard | 196 - org.opendaylight.controller.sal-clustering-commons - 1.5.2.SNAPSHOT | member-3-shard-default-operational: Applying state for log index 1940 data org.opendaylight.controller.cluster.datastore.persisted.PurgeTransactionPayload@74db3f62<br/> 2017-08-21 22:50:10,362 | DEBUG | rd-dispatcher-34 | Shard | 196 - org.opendaylight.controller.sal-clustering-commons - 1.5.2.SNAPSHOT | member-3-shard-default-operational (Leader): Setting last applied to 1939<br/> 2017-08-21 22:50:10,362 | DEBUG | rd-dispatcher-34 | Shard | 196 - org.opendaylight.controller.sal-clustering-commons - 1.5.2.SNAPSHOT | member-3-shard-default-operational: Moving last applied index from 1940 to 1939<br/> 2017-08-21 22:50:10,362 | DEBUG | rd-dispatcher-34 | Shard | 196 - org.opendaylight.controller.sal-clustering-commons - 1.5.2.SNAPSHOT | member-3-shard-default-operational: Applying state for log index 1939 data org.opendaylight.controller.cluster.datastore.persisted.CommitTransactionPayload@1cb3e6f2<br/> 2017-08-21 22:50:10,362 | DEBUG | rd-dispatcher-34 | ShardDataTree | 203 - org.opendaylight.controller.sal-distributed-datastore - 1.5.2.SNAPSHOT | member-3-shard-default-operational: Applying foreign transaction member-3-datastore-operational-fe-1-txn-1133-0<br/> 2017-08-21 22:50:10,362 | DEBUG | rd-dispatcher-34 | Shard | 196 - org.opendaylight.controller.sal-clustering-commons - 1.5.2.SNAPSHOT | member-3-shard-default-operational (Leader): Setting last applied to 1940<br/> 2017-08-21 22:50:10,362 | DEBUG | rd-dispatcher-34 | Shard | 196 - org.opendaylight.controller.sal-clustering-commons - 1.5.2.SNAPSHOT | member-3-shard-default-operational: Moving last applied index from 1939 to 1940<br/> 2017-08-21 22:50:10,362 | DEBUG | rd-dispatcher-34 | Shard | 196 - org.opendaylight.controller.sal-clustering-commons - 1.5.2.SNAPSHOT | member-3-shard-default-operational: Applying state for log index 1940 data org.opendaylight.controller.cluster.datastore.persisted.PurgeTransactionPayload@74db3f62</p> <p>Alternative fix: <a href="https://git.opendaylight.org/gerrit/62169" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/62169</a></p> <p>The alternative fix should go in anyway, because the API contract should follow what Akka Persistence does.</p> <p>In case of a sync call, AP juggles the actor mailbox, stashing any non-persistence-related messages and once it's done persisting and will involve the callback.</p> <p>In case of an async call, AP continues normal operation, scheduling a message in the actor to invoke the callback.</p> <p>NonPersistentDataProvider invokes both cases directly in the caller.</p> <p>The patch teaches NonPersistentDataProvider to send an internal message within the actor (which is an actor capability expressed as an interface), which it does in case of an async call.</p> <p>Since the configuration data store does not seem to be exhibiting this (or we are not hammering it hard enough), aligning the behavior should make the bug go away.</p> <p>If this is not the call site causing trouble, <a href="https://git.opendaylight.org/gerrit/62170" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/62170</a> adds stack traces to lastApplied transitions, which should pinpoint the problem.</p> <p>A patch was submitted to revert the changes and fix this bug in Carbon SR2:</p> <p><a href="https://git.opendaylight.org/gerrit/#/c/62169/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/62169/</a></p> <p>To better assess the impact of this bug and fix, could someone from your team please help us identify the following:<br/> Regression: Is this bug a regression of functionality/performance/feature compared to Carbon?<br/> Severity: Could you elaborate on the severity of this bug? Is this a BLOCKER such that we cannot release Carbon SR2 without it?<br/> Workaround: Is there a workaround such that we can write a release note instead?<br/> Testing: Could you also elaborate on the testing of this patch? How extensively has this patch been tested? Is it covered by any unit tests or system tests? <br/> Impact: Does this fix impact any dependent projects?</p> <p>I think the NetVirt side is good with this change. Jobs <span class="error">&#91;2&#93;</span> an <span class="error">&#91;3&#93;</span> below look good.</p> <p>Sometimes there are random failures where a node does not come back properly such as in job <span class="error">&#91;4&#93;</span>. We try to bring ODL1 back into the cluster but it fails to come back within 5 minutes. Then we move to the next tests and they fail. That ODL1 is hitting the below issue. Is there anything we can do to get past that? Is this bug-9006?</p> <p>2017-08-25 02:02:38,430 | WARN | saction-32-34'}} | DeadlockMonitor | 126 - org.opendaylight.controller.config-manager - 0.6.2.SNAPSHOT | ModuleIdentifier</p> {factoryName='runtime-generated-mapping', instanceName='runtime-mapping-singleton'} <p> did not finish after 284864 ms</p> <p><span class="error">&#91;2&#93;</span> <a href="https://jenkins.opendaylight.org/releng/user/shague/my-views/view/3node/job/netvirt-csit-3node-openstack-ocata-gate-stateful-carbon/25/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/user/shague/my-views/view/3node/job/netvirt-csit-3node-openstack-ocata-gate-stateful-carbon/25/</a></p> <p><span class="error">&#91;3&#93;</span> <a href="https://jenkins.opendaylight.org/releng/user/shague/my-views/view/3node/job/netvirt-csit-3node-openstack-ocata-upstream-stateful-carbon/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/user/shague/my-views/view/3node/job/netvirt-csit-3node-openstack-ocata-upstream-stateful-carbon/</a></p> <p>4] <a href="https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-3node-openstack-ocata-gate-stateful-carbon/24/log.html.gz#s1-s1-t13-k2-k2-k8" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-3node-openstack-ocata-gate-stateful-carbon/24/log.html.gz#s1-s1-t13-k2-k2-k8</a></p> <p>Robert, Sam, should this BZ issue be closed now, or is there anything left here?</p> <p>Regression: Is this bug a regression of functionality/performance/feature compared to Carbon?</p> <p>Yes. This is a pre-existing issue, which was exposed by <a href="https://jira.opendaylight.org/browse/NETVIRT-673" title="Traffic between a VM without FIP and a VM with FIP is not working" class="issue-link" data-issue-key="NETVIRT-673"><del>NETVIRT-673</del></a> which itself is a regression.</p> <p>Severity: Could you elaborate on the severity of this bug? Is this a BLOCKER such that we cannot release Carbon SR2 without it?</p> <p>It is, as operational datastore can easily encounter errors, which will resurface even after application-level recovery. Data corruption is possible, too.</p> <p>Workaround: Is there a workaround such that we can write a release note instead?</p> <p>There is no feasible workaround.</p> <p>Testing: Could you also elaborate on the testing of this patch? How extensively has this patch been tested? Is it covered by any unit tests or system tests? </p> <p>Tested with netvirt CSIT.</p> <p>Impact: Does this fix impact any dependent projects?</p> <p>Aside from fixing issues, there is no impact as far as we know.</p> Blocks CONTROLLER-1756 CONTROLLER-1762 CONTROLLER-1763 MDSAL-275 Duplicate CONTROLLER-1761 Development External issue ID 9028 External issue URL Issue Type ODL SR Target Milestone Rank 0|i02sj3: