[CONTROLLER-1954] Lower akka-segmented-journal memory footprint Created: 10/Jul/20 Updated: 06/May/21 Resolved: 06/May/21 |
|
| Status: | Resolved |
| Project: | controller |
| Component/s: | clustering |
| Affects Version/s: | None |
| Fix Version/s: | 4.0.0, 3.0.8 |
| Type: | Improvement | Priority: | High |
| Reporter: | Robert Varga | Assignee: | Robert Varga |
| Resolution: | Done | Votes: | 0 |
| Labels: | pt | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Description |
|
Heap analysis shows that io.atomix.storage.journal.FileChannelJournalSegmentWriter ends up allocating HeapByteBuffers with twice the size of maximum entry size. In our default full install, this means ODL ends up eating 306MiB of heap off the bat. We also have a problem where the maximum entry size is limited, whereas we can have rather huge entries stored in the journal. Introduce a revised journal version, which will seamlessly migrate when attached, so that we perform entry chunking – i.e. lower the maximum entry to something around 128KiB and split larger Akka journal entries into multiple file entries. |
| Comments |
| Comment by Robert Varga [ 25/Feb/21 ] |
|
Rather than increasing complexity, let's just switch to memory-mapped files by default, which solves the heap problem. |