https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/CONTROLLER-1297 controller <p>The following error was seen after a controller restart (Helium SR2):</p> <p>java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: Metadata not available for modification [NodeModification [identifier=(com:brocade:neutron:odl?revision=2014-10-02)subnet, modificationType=SUBTREE_MODIFIED, childModification={(com:brocade:neutron:odl?revision=2014-10-02)subnet[</p> {(com:brocade:neutron:odl?revision=2014-10-02)id=ace19864-b874-47a9-9cef-b02afd52f37b} <p>]=NodeModification [identifier=(com:brocade:neutron:odl?revision=2014-10-02)subnet[</p> {(com:brocade:neutron:odl?revision=2014-10-02)id=ace19864-b874-47a9-9cef-b02afd52f37b} <p>], modificationType=DELETE, childModification={}]}]]<br/> at java.util.concurrent.FutureTask.report(FutureTask.java:122)<span class="error">&#91;:1.7.0_76&#93;</span><br/> at java.util.concurrent.FutureTask.get(FutureTask.java:188)<span class="error">&#91;:1.7.0_76&#93;</span><br/> at org.opendaylight.controller.cluster.datastore.Shard.syncCommitTransaction(Shard.java:586)<span class="error">&#91;301:org.opendaylight.controller.sal-distributed-datastore:1.1.2.Helium-SR2&#93;</span><br/> at org.opendaylight.controller.cluster.datastore.Shard.onRecoveryComplete(Shard.java:729)<span class="error">&#91;301:org.opendaylight.controller.sal-distributed-datastore:1.1.2.Helium-SR2&#93;</span><br/> at org.opendaylight.controller.cluster.raft.RaftActor.onRecoveryCompletedMessage(RaftActor.java:257)<span class="error">&#91;294:org.opendaylight.controller.sal-akka-raft:1.1.2.Helium-SR2&#93;</span><br/> at org.opendaylight.controller.cluster.raft.RaftActor.handleRecover(RaftActor.java:160)<span class="error">&#91;294:org.opendaylight.controller.sal-akka-raft:1.1.2.Helium-SR2&#93;</span><br/> at org.opendaylight.controller.cluster.common.actor.AbstractUntypedPersistentActor.onReceiveRecover(AbstractUntypedPersistentActor.java:52)<span class="error">&#91;293:org.opendaylight.controller.sal-clustering-commons:1.1.2.Helium-SR2&#93;</span><br/> at org.opendaylight.controller.cluster.datastore.Shard.onReceiveRecover(Shard.java:237)<span class="error">&#91;301:org.opendaylight.controller.sal-distributed-datastore:1.1.2.Helium-SR2&#93;</span></p> <p>The modification is for a node delete and it seems "Metadata not available ..." indicates the node doesn't exist. If that's true, how did this modification entry get into the persisted journal? Transaction modifications should only get into the journal if the transaction succeeds.</p> <p>The ramification of this failure is that the rest of the data failed to recover as well. This is b/c we batch journal entries 5000 at a time into a single transaction. This is more performant but the side effect is that one failed modification fails everything.</p> <p>In addition, the failed entry remains in the RaftActor's in-memory journal so, in a 3 node cluster, if it becomes the leader then it wipes out the other nodes too. We need to protect against a corrupted journal (or a recovery failure) on one node from corrupting the whole cluster.</p> <p>Operating System: All<br/> Platform: All</p> CONTROLLER-1297

Clustering: Journal recovery error on restart

Bug Resolved Done Tom Pantelis Tom Pantelis Thu, 7 May 2015 23:19:43 +0000 Tue, 25 Aug 2015 15:42:32 +0000 Tue, 25 Aug 2015 15:42:32 +0000 Helium clustering 0 3 <p>Interestingly, changing it to apply the journal modification entries one at a time, as was done when they were originally committed, alleviates the issue - no errors and all the data was recovered.</p> <p>So it seems there's an issue when certain modification sequences are applied all at once rather than one by one. Theoretically, both ways should yield the same result.<br/> I suspect this issue doesn't exist in Li as the tree modification code was refactored and there were bug fixes. Will verify to make sure.</p> <p>Some of my colleagues have reported a problem where in a single transaction they have seen this problem. I'll try to find the exact sequence and let you know. I think this problem could be happening due that reason.</p> <p>To summarize,</p> <p>This works,</p> <p>tx1.put(id, node);<br/> tx1.submit() </p> <p>tx2.merge(id, node);<br/> tx2.submit();</p> <p>This does not</p> <p>tx3.put(id, node);<br/> tx3.merge(id, node);<br/> tx3.submit();</p> <p>Submitted <a href="https://git.opendaylight.org/gerrit/#/c/20736/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/20736/</a> to stable/helium to defauklt the journal recovery batch size to 1.</p> <p>May be fixed. Appears to work in Lithium.</p> Development External issue ID 3154 External issue URL ODL SR Target Milestone Priority Rank 0|i02ppb: