feat(search): Multishard cutoffs #1924
Conversation
dranikpg
force-pushed
the
multishard-search
branch
from
24b2025
to
fe7559a
Compare
|
My condolences to the one reviewing it 🙂 |
I actually wrote Vlad - Kostas 1-1 only to not submit the comment because it would have been noise. Anyway, when you wrote this I could not resist not to write this msg 😛 I would argue mine was simpler because it was mostly boilerplate (I haven't looked yours TBH so it could be the same case) so it's not really 1-1 and probably you are getting the 👑 👨🍳 |
dranikpg
force-pushed
the
multishard-search
branch
from
fe7559a
to
c94d9ad
Compare
dranikpg
force-pushed
the
multishard-search
branch
from
c94d9ad
to
30ea6b8
Compare
dranikpg
force-pushed
the
multishard-search
branch
from
30ea6b8
to
883d326
Compare
dranikpg
changed the title
|
Please take a quick look at this. There is still some polishment left but the general "structure" is ready. I'd wish for this to be merged before we announce search because it's important for fast multishard queries PS: Yes there is lots of code and it's complicated 🙂 I'd start looking from MultiShardSearch::Run because it's the main entry point w everything else being tailored to fit it's needs |
dranikpg
force-pushed
the
multishard-search
branch
from
883d326
to
7755b92
Compare
| size_t avg_shard_min = hits * intlog2(hits) / (12 + shard_set->size() / 10); | ||
| avg_shard_min -= min(avg_shard_min, min(hits, size_t(5))); |
There was a problem hiding this comment.
Can you explain the rationale here?
There was a problem hiding this comment.
Experimentally from #1892 😄 Those formulas might not be final
There was a problem hiding this comment.
I'd link to that, at least until you change it :)
| // If all shards have at least avg min, keep the bare minimum needed to cover the request | ||
| size_t limit = requested / shards + 1; | ||
|
|
||
| // Aggregations like SORTBY and KNN reorder the result and thus introduce some variance |
There was a problem hiding this comment.
Don't we have to fetch all matching for SORTBY?
There was a problem hiding this comment.
No/Yes 🤓 We have to fetch all, but not serialize all. That is what the doc reference is for. If we find out when building the reply that some entries should have been included but are not serialized, we'll refill (see BuildSortedOrder())
Signed-off-by: Vladislav Oleshko <[email protected]>
Signed-off-by: Vladislav Oleshko <[email protected]>
Signed-off-by: Vladislav Oleshko <[email protected]>
dranikpg
force-pushed
the
multishard-search
branch
from
8fe00de
to
52856f4
Compare
No description provided.