As noted in CONTROLLER-2043, entry indexing does a rather poor job in handling its memory buffer. During indexing it specifically does this at the end of the loop iteration:

        // Read more bytes from the segment if necessary.
        if (memory.remaining() < maxEntrySize) {
          memory.clear();
          channel.position(position);
          channel.read(memory, position);
          memory.flip();
        }

        memory.mark();
        length = memory.getInt();

What this means is that it discards the buffer if there is even a slight chance that it holds an incomplete entry. The above condition holds true for all entries stored in the last maxEntrySize bytes of a file – which in our case can be thousands of entries!

The logic here needs to be improved so that:

1. we check to see if there are at least 4 bytes available (to satisfy reading of the header)
2. check and validate the entry size to see if the entry checksum and body is in the memory
3. if it turns out we need more data, first run memory.compact() and then read up to the buffer's capacity

The end result should be that we never throw away bytes we have read and always fill the buffer its limit, if possible.