https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/CONTROLLER-1328 controller <p>Steps to reproduce<br/> 1. Create an OFP distribution on master<br/> 2. Start the controller install odl-openflowplugin-all<br/> 3. Connect mininet - sudo mn --controller=remote,ip=&lt;host&gt; --topo tree,3<br/> 4. Run config flow blaster a couple of times<br/> ./flow_config_blaster.py --auth --threads 5 --flows 5000 --no-delete<br/> 5. Check the number of flows in config and note the number<br/> 5. Stop controller gracefully.<br/> 6. Stop mininet<br/> 7. Start the controller<br/> 8. Connect mininet<br/> 9. Check the number of flows in config. Around 400 are missing.</p> <p>The log is filled with messages : </p> <p>2015-05-18 14:07:20,796 | INFO | lt-dispatcher-25 | Shard | 175 - org.opendaylight.controller.sal-akka-raft - 1.2.0.SNAPSHOT | Recovery snapshot applied for member-1-shard-inventory-config in 2.119 s: snapshotIndex=39999, snapshotTerm=1, journal-size=0<br/> 2015-05-18 14:07:20,805 | INFO | lt-dispatcher-25 | Shard | 175 - org.opendaylight.controller.sal-akka-raft - 1.2.0.SNAPSHOT | member-1-shard-inventory-config: Received ReplicatedLogEntry for recovery: index: 40322, size: 1445<br/> 2015-05-18 14:07:20,806 | INFO | lt-dispatcher-25 | Shard | 175 - org.opendaylight.controller.sal-akka-raft - 1.2.0.SNAPSHOT | member-1-shard-inventory-config: Received ReplicatedLogEntry for recovery: index: 40323, size: 1441<br/> 2015-05-18 14:07:20,806 | INFO | lt-dispatcher-25 | Shard | 175 - org.opendaylight.controller.sal-akka-raft - 1.2.0.SNAPSHOT | member-1-shard-inventory-config: Received ReplicatedLogEntry for recovery: index: 40324, size: 1441<br/> 2015-05-18 14:07:20,806 | INFO | lt-dispatcher-25 | Shard | 175 - org.opendaylight.controller.sal-akka-raft - 1.2.0.SNAPSHOT | member-1-shard-inventory-config: Received ReplicatedLogEntry for recovery: index: 40325, size: 1445<br/> 2015-05-18 14:07:20,807 | INFO | lt-dispatcher-25 | Shard | 175 - org.opendaylight.controller.sal-akka-raft - 1.2.0.SNAPSHOT | member-1-shard-inventory-config: Received apply journal entries for recovery, applying to state: 40000 to 40322<br/> 2015-05-18 14:07:20,813 | ERROR | lt-dispatcher-25 | Shard | 175 - org.opendaylight.controller.sal-akka-raft - 1.2.0.SNAPSHOT | Log entry not found for index 40004<br/> 2015-05-18 14:07:20,813 | INFO | lt-dispatcher-25 | Shard | 175 - org.opendaylight.controller.sal-akka-raft - 1.2.0.SNAPSHOT | member-1-shard-inventory-config: Received apply journal entries for recovery, applying to state: 40004 to 40323<br/> 2015-05-18 14:07:20,814 | ERROR | lt-dispatcher-25 | Shard | 175 - org.opendaylight.controller.sal-akka-raft - 1.2.0.SNAPSHOT | Log entry not found for index 40004<br/> 2015-05-18 14:07:20,814 | INFO | lt-dispatcher-25 | Shard | 175 - org.opendaylight.controller.sal-akka-raft - 1.2.0.SNAPSHOT | member-1-shard-inventory-config: Received apply journal entries for recovery, applying to state: 40004 to 40324<br/> 2015-05-18 14:07:20,814 | ERROR | lt-dispatcher-25 | Shard | 175 - org.opendaylight.controller.sal-akka-raft - 1.2.0.SNAPSHOT | Log entry not found for index 40004<br/> 2015-05-18 14:07:20,814 | INFO | lt-dispatcher-25 | Shard | 175 - org.opendaylight.controller.sal-akka-raft - 1.2.0.SNAPSHOT | member-1-shard-inventory-config: Received apply journal entries for recovery, applying to state: 40004 to 40325<br/> 2015-05-18 14:07:20,814 | ERROR | lt-dispatcher-25 | Shard | 175 - org.opendaylight.controller.sal-akka-raft - 1.2.0.SNAPSHOT | Log entry not found for index 40004<br/> 2015-05-18 14:07:20,814 | INFO | lt-dispatcher-25 | Shard | 175 - org.opendaylight.controller.sal-akka-raft - 1.2.0.SNAPSHOT | member-1-shard-inventory-config: Received ReplicatedLogEntry for recovery: index: 40326, size: 1441<br/> 2015-05-18 14:07:20,814 | INFO | lt-dispatcher-25 | Shard | 175 - org.opendaylight.controller.sal-akka-raft - 1.2.0.SNAPSHOT | member-1-shard-inventory-config: Received apply journal entries for recovery, applying to state: 40004 to 40326<br/> 2015-05-18 14:07:20,814 | ERROR | lt-dispatcher-25 | Shard | 175 - org.opendaylight.controller.sal-akka-raft - 1.2.0.SNAPSHOT | Log entry not found for index 40005<br/> 2015-05-18 14:07:20,814 | INFO | lt-dispatcher-25 | Shard | 175 - org.opendaylight.controller.sal-akka-raft - 1.2.0.SNAPSHOT | member-1-shard-inventory-config: Received ReplicatedLogEntry for recovery: index: 40327, size: 1445</p> <p>The snapshot was taken at index=39999 with 0 unapplied entries. <br/> The persistent journal starts with the index=40322. <br/> Where did the 322 logs go?</p> <p>Operating System: All<br/> Platform: All</p> CONTROLLER-1328

Clustering: Recovery misses flows installed

Bug Resolved Done Kamal Rameshan Kamal Rameshan Mon, 18 May 2015 21:21:13 +0000 Tue, 2 Jun 2015 15:02:40 +0000 Tue, 2 Jun 2015 15:02:40 +0000 Post-Helium clustering 0 2 <p>Attachment recovery-issue-1.log has been added with description: recovery log</p> <p>Sanpshots are taken with 0 unapplied entries at batches of 20000</p> <p>2015-05-18 14:02:42,024 | INFO | lt-dispatcher-33 | Shard | 175 - org.opendaylight.controller.sal-akka-raft - 1.2.0.SNAPSHOT | member-1-shard-inventory-config: Persisting of snapshot done:Snapshot=</p> {lastTerm:1, lastIndex:39999, LastAppliedIndex:39999, LastAppliedTerm:1, UnAppliedEntries size:0} <p>2015-05-18 14:02:42,025 | INFO | lt-dispatcher-33 | Shard | 175 - org.opendaylight.controller.sal-akka-raft - 1.2.0.SNAPSHOT | member-1-shard-inventory-config: Removed in-memory snapshotted entries, adjusted snaphsotIndex:39999 and term:1<br/> 2015-05-18 14:02:42,132 | INFO | lt-dispatcher-21 | Shard | 175 - org.opendaylight.controller.sal-akka-raft - 1.2.0.SNAPSHOT | member-1-shard-inventory-config: SaveSnapshotSuccess received for snapshot</p> <p>The "Received ReplicatedLogEntry for recovery" output is logged to DEBUG. I assume you changed it to INFO in your build.</p> <p>I suspect when we trimmed the persistent journal after the snapshot we blew away the 322 journal entries that occurred in between the time the snapshot was started and it was committed. So something's wrong there. Either the lastSequenceNumber to delete was incorrect or the 322 entries should've been in the unapplied list.</p> <p>Yes i did change the log levels to info (somehow logback.xml change was not working for me)</p> <p>Am trying to dig in...</p> <p>I've never used logback.xml - I think that's legacy. With karaf, you turn on debug in etc/org.pos4.pax.logging.cfg. It's standard log4j, e.g.</p> <p> log4j.logger.org.opendaylight...Shard=DEBUG</p> <p>I'll take a look at the code as well.</p> <p><a href="https://git.opendaylight.org/gerrit/#/c/21372/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/21372/</a></p> Development External issue ID 3260 External issue URL Issue Type ODL SR Target Milestone Priority Rank 0|i02pw7: