refactor(pageserver): better k-merge implementation for tiered compaction

[skyzh]

Problem

The original k-merge implementation might be buggy and is hard to reason about. This pull request rewrites the k-merge implementation to make it easier to read and potentially avoids the bug that keys are not ordered.

ref #7703

Summary of changes

Adds LayerIterator and MergeIterator. They provide key_lsn, next, and is_end APIs.

Checklist before requesting a review

• I have performed a self-review of my code.
• If it is a core feature, I have added thorough tests.
• Do we need to implement analytics? if so did you add the relevant metrics to the dashboard?
• If this PR requires public announcement, mark it with /release-notes label and add several sentences in this section.

Checklist before merging

• Do not forget to reformat commit message to not include the above checklist

[github-actions]

No tests were run or test report is not available

Test coverage report is not available

_{The comment gets automatically updated with the latest test results
a40eee3 at 2024-05-21T15:23:36.826Z :recycle:}

[arpad-m]

There is a reproducer for the assertion in #7758, but it doesn't reproduce it all the time. Could you try running that test a couple of times (maybe 10 times) to see if the error is really gone?

As for the next function, it doesn't return an Option. Ideally, it should return either return Option<Result<...>> or Result<Option<...>,...>. Returning None once an iterator has finished is a convention in Rust and it helps avoid bugs where you invoke "is at end" check function in a wrong manner.

[hlinnaka]

Before a layer is loaded, the layer's key_lsn() is the key, lsn from the layer's starting key. But after loading, it's the key,lsn of the first key-value pair that's actually in the layer. That's not necessarily the same thing! In particular, with L0 layers, the start of the layer's key range is always 0, but the first key-value pair almost certainly has a higher key.

Consider the following simplified example. I'm leaving out the LSNs, they are not important for this. Imagine you're merging two layers:

Layer A: key range 0-10. It contains a single key, 8
Layer B: key range 5-15. It contains a single key, 5

The code as written would incorrectly return keys 8, 5.

This function is clearly a more tricky it seems at first, so maybe we should move the code to a separate source file, and also add some unit tests with some simple mock layer implementation, with examples like the above.

[skyzh]

well, at least it's easier to write complex logic in async fn instead of manually maintain the state machine for async stream 😛 will add some test cases tmr

[arpad-m]

Before a layer is loaded, the layer's key_lsn() is the key, lsn from the layer's starting key. But after loading, it's the key,lsn of the first key-value pair that's actually in the layer. That's not necessarily the same thing!

yup the old code beautifully handles this case: if it finds an unloaded layer as the first one, it only means that there is the potential to yield the first element from it. It puts the just loaded layer back into the heap, and now the order depends on the first actual element.

[skyzh]

I've been looking at the previous k-merge algorithm and couldn't find anything wrong. Put this PR on hold and focus on solving other issues first.